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(54) COMMUNICATION EQUIPMENT AND METHOD THEREFOR, COMMUNICATION 
SYSTEM, RECORDING MEDIUM AND PROGRAM 

(57)Abstract: 
PROBLEM TO BE SOLVED: To cipher images. 

SOLUTION: In a diagram 135, the image for which the image of an object in a black 
round shape is picked up is indicated in the utmost left column top row of a matrix. 
From the state, for instance, at the time of adding a movement blur in the vertical 
direction, the object in the black round shape becomes the image in which the 
movement blur is generated in the vertical direction as indicated in the center column 



top row. Further, at the time of generating the movement blur in the horizontal 
direction, it becomes the image in which the movement blur is generated in the 
vertical and horizontal directions of the object as indicated in the center column 
center row. By repeating similar processings, the possibility of deciphering the object 
is lowered and thus, the image itself can be ciphered. 

* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect 
the original precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1]A communication apparatus comprising: 

A demand information input means which inputs a user's demand information. 

A foreground ingredient picture which consists of a foreground object ingredient which 

constitutes a foreground object of image data which consists of a pixel value 

determined according to quantity of light which constitutes a picture with which it 

integrated in time for every pixel according to said demand information. 

A synthesizing means which compounds a background component image which 

consists of a background object component which constitutes background objects of 

said image data, and generates image composing. 

An image composing output means which outputs image composing generated by said 
synthesizing means. 

[Claim 2]The communication apparatus according to claim 1, wherein said image 
composing output means outputs said image composing to said user's communication 
apparatus. 

[Claim 3]Said demand information input means with a user's demand information Said 
predetermined foreground ingredient picture, Input significant information used when 
compounding said predetermined background component image, and said 
predetermined foreground ingredient picture and said predetermined background 



component image, and said synthesizing means, The communication apparatus 
according to claim 1 compounding a foreground ingredient picture predetermined 
[ said ] inputted with said demand information by said demand information input means, 
and said predetermined background component image based on said significant 
information, and generating image composing. 

[Claim 4]Said demand information input means as significant information used when 
compounding said predetermined foreground ingredient picture, said predetermined 
background component image, and said predetermined foreground ingredient picture 
and said predetermined background component image with a users demand 
information, Input the mixture ratio of a mixing zone of said predetermined foreground 
ingredient picture and said predetermined background component image, and said 
synthesizing means, The communication apparatus according to claim 3 compounding 
a foreground ingredient picture predetermined [ said ] inputted with said demand 
information by said demand information input means, and said predetermined 
background component image based on said mixture ratio as said significant 
information, and generating image composing. 

[Claim 5]Said demand information input means as significant information used when 
compounding said predetermined foreground ingredient picture, said predetermined 
background component image, and said predetermined foreground ingredient picture 
and said predetermined background component image with a user's demand 
information, Input movement quantity and the motion direction of said foreground 
ingredient picture, and said synthesizing means, Based on said movement quantity and 
said motion direction of as said significant information, move said predetermined 
foreground ingredient picture inputted with said demand information by said demand 
information input means, and a Japanese quince is adjusted, The communication 
apparatus according to claim 3 compounding with said predetermined background 
component image, and generating image composing. 

[Claim 6]Said demand information input means as significant information used when 
compounding said predetermined foreground ingredient picture, said predetermined 
background component image, and said predetermined foreground ingredient picture 
and said predetermined background component image with a user's demand 
information, Input initial position information, movement quantity, and the motion 
direction of said foreground ingredient picture, and said synthesizing means, Based on 
said movement quantity and said motion direction of as said significant information, 
move said predetermined foreground ingredient picture inputted with said demand 
information by said demand information input means, adjust a Japanese quince, and 
said mixture ratio is computed, A foreground ingredient picture predetermined [ said ] 
with which motion dotage was adjusted, and said predetermined background 
component image with said computed mixture ratio. The communication apparatus 
according to claim 3 compounding based on initial position information, said movement 



quantity, and said motion direction of said foreground ingredient picture of as 
significant information, and generating image composing. 

[Claim 7]A foreground ingredient image identifier from which said demand information 
input means discriminates said predetermined foreground ingredient picture with a 
user's demand information, Input a background component image identifier which 
identifies said predetermined background component image, and said significant 
information, and said synthesizing means, A foreground ingredient picture 
corresponding to said foreground ingredient image identifier according to a foreground 
ingredient image identifier which identifies said predetermined foreground ingredient 
picture with said demand information, a background component image identifier which 
identifies said predetermined background component image, and said significant 
information, The communication apparatus according to claim 1 compounding a 
background component image corresponding to said background component image 
identifier based on said significant information, and generating image composing. 
[Claim 8]The communication apparatus according to claim 1 having further a charging 
means which performs accounting according to said demand information. 
[Claim 9]According to said demand information, said charging means Said user's 
identifier, an identifier of said communication apparatus, And the communication 
apparatus according to claim 8 generating accounting information including amount 
information corresponding to said demand information, and performing accounting to 
said user's financial account based on said accounting information. 
[Claim 10]Have further a point storage means which memorizes a point equivalent to 
cash for every user used for said accounting, and said charging means, According to 
said demand information, said user's identifier, an identifier of said communication 
apparatus, And the communication apparatus according to claim 8 performing 
accounting when only a part corresponding to said amount information subtracts a 
point size which generated accounting information including amount information 
corresponding to said image composing, and was memorized by said point storage 
means. 

[Claim 1 1]The communication apparatus according to claim 8 after said output means 
is completed [ accounting ] by said charging means, wherein it outputs said image 
composing to said user's communication apparatus which accounting completed. 
[Claim 12]A correspondence procedure comprising: 

A demand information inputting step which inputs a user's demand information. 

A foreground ingredient picture which consists of a foreground object ingredient which 

constitutes a foreground object of image data which consists of a pixel value 

determined according to quantity of light which constitutes a picture with which it 

integrated in time for every pixel according to said demand information. 

A synthetic step which compounds a background component image which consists of 

a background object component which constitutes background objects of said image 



data, and generates image composing. 

An image composing output step which outputs image composing generated by 
processing of said synthetic step. 

[Claim 13]A recording medium with which a program which a computer can read is 
recorded, comprising: 

A demand information inputting control step which controls an input of a user's 
demand information. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to said demand information. 
Composition of a background component image which consists of a background object 
component which constitutes background objects of said image data. 
A synthetic control step which controls generation of image composing, and an image 
composing output-control step which controls said output of image composing by 
which synthetic control step generation was carried out. 

[Claim 14]According to a demand information inputting control step which controls an 
input of a user's demand information, and said demand information, A foreground 
ingredient picture which consists of a foreground object ingredient which constitutes 
a foreground object of image data which consists of a pixel value determined 
according to quantity of light which constitutes a picture with which it integrated in 
time for every pixel, Composition of a background component image which consists of 
a background object component which constitutes background objects of said image 
data, A program which makes a computer perform a synthetic control step which 
controls generation of image composing, and an image composing output-control step 
which controls said output of image composing by which synthetic control step 
generation was carried out. 

[Claim 15]A communications system which consists of the 1st communication 
apparatus and 2nd communication apparatus, comprising: 

A demand information input means as which said 1st communication apparatus inputs 
a user's demand information. 

A demand information transmission means which transmits demand information 
inputted by said demand information input means to said 2nd communication 
apparatus. 

A demand information receiving means which receives said demand information which 
is equipped with an image composing reception means which receives image 
composing transmitted from said 2nd communication apparatus according to said 
demand information, and to which said 2nd communication apparatus is transmitted 



from said 1st communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to said demand information. 
A synthesizing means which compounds a background component image which 
consists of a background object component which constitutes background objects of 
said image data, and generates said image composing. 

An image composing transmitting means which transmits said image composing 
generated by said synthesizing means to said 1st communication apparatus. 

[Claim 16]Said demand information input means with a user's demand information Said 
predetermined foreground ingredient picture, Input significant information used when 
compounding said predetermined background component image, and said 
predetermined foreground ingredient picture and said predetermined background 
component image, and said synthesizing means, The communication apparatus 
according to claim 15 compounding a foreground ingredient picture predetermined 
[ said ] inputted with said demand information by said demand information input means, 
and said predetermined background component image based on said significant 
information, and generating image composing. 

[Claim 17]Said demand information input means as significant information used when 
compounding said predetermined foreground ingredient picture, said predetermined 
background component image, and said predetermined foreground ingredient picture 
and said predetermined background component image with a user's demand 
information, Input the mixture ratio of a mixing zone of said predetermined foreground 
ingredient picture and said predetermined background component image, and said 
synthesizing means, The communication apparatus according to claim 16 
compounding a foreground ingredient picture predetermined [ said ] inputted with said 
demand information by said demand information input means, and said predetermined 
background component image based on said mixture ratio as said significant 
information, and generating image composing. 

[Claim 18]Said demand information input means as significant information used when 
compounding said predetermined foreground ingredient picture, said predetermined 
background component image, and said predetermined foreground ingredient picture 
and said predetermined background component image with a user's demand 
information, Input movement quantity and the motion direction of said foreground 
ingredient picture, and said synthesizing means, Based on said movement quantity and 
said motion direction of as said significant information, move said predetermined 
foreground ingredient picture inputted with said demand information by said demand 
information input means, and a Japanese quince is adjusted, The communication 



apparatus according to claim 16 compounding with said foreground ingredient picture 
and generating image composing. 

[Claim 19]Said demand information input means as significant information used when 
compounding said predetermined foreground ingredient picture, said predetermined 
background component image, and said predetermined foreground ingredient picture 
and said predetermined background component image with a user's demand 
information, Input initial position information, movement quantity, and the motion 
direction of said foreground ingredient picture, and said synthesizing means, Based on 
said movement quantity and said motion direction of as said significant information, 
move said predetermined foreground ingredient picture inputted with said demand 
information by said demand information input means, adjust a Japanese quince, and 
said mixture ratio is computed, A foreground ingredient picture predetermined [ said ] 
with which motion dotage was adjusted, and said predetermined background 
component image with said computed mixture ratio. The communication apparatus 
according to claim 16 compounding based on initial position information, said 
movement quantity, and said motion direction of said foreground ingredient picture of 
as significant information, and generating image composing. 

[Claim 20]A foreground ingredient image identifier from which said demand information 
input means discriminates said predetermined foreground ingredient picture with a 
user's demand information, Input a background component image identifier which 
identifies said predetermined background component image, and said significant 
information, and said synthesizing means, A foreground ingredient picture 
corresponding to said foreground ingredient image identifier according to a foreground 
ingredient image identifier which identifies said predetermined foreground ingredient 
picture with said demand information, a background component image identifier which 
identifies said predetermined background component image, and said significant 
information, The communication apparatus according to claim 16 compounding a 
background component image corresponding to said background component image 
identifier based on said significant information, and generating image composing. 
[Claim 21]The communication apparatus according to claim 15 having further a 
charging means which performs accounting according to said demand information. 
[Claim 22]According to said demand information, said charging means Said user's 
identifier, an identifier of said communication apparatus, And the communication 
apparatus according to claim 21 generating accounting information including amount 
information corresponding to said demand information, and performing accounting to 
said user's financial account based on said accounting information. 
[Claim 23]Have further a point storage means which memorizes a point for every user 
used for said accounting, and said charging means, According to said demand 
information, said user's identifier, an identifier of said communication apparatus, And 
the communication apparatus according to claim 21 performing accounting when only 



a part corresponding to said amount information subtracts a point size which 

generated accounting information including amount information corresponding to said 

image composing, and was memorized by said point storage means. 

[Claim 24]The communication apparatus according to claim 21 after said output 

means is completed [ accounting ] by said charging means, wherein it outputs said 

image composing to said user's communication apparatus which accounting 

completed. 

[Claim 25]A correspondence procedure of a communications system which consists 
of the 1st communication apparatus and 2nd communication apparatus characterized 
by comprising the following. 

A demand information inputting step into which a correspondence procedure of said 

1st communication apparatus inputs a user's demand information. 

A demand transmitting information step which transmits demand information inputted 

by processing of said demand information inputting step to said 2nd communication 

apparatus. 

A demand information reception step which receives said demand information to 
which a correspondence procedure of said 2nd communication apparatus is 
transmitted from said 1st communication apparatus including an image composing 
receiving step which receives image composing transmitted from said 2nd 
communication apparatus according to said demand information. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to said demand information. 
A synthetic step which compounds a background component image which consists of 
a background object component which constitutes background objects of said image 
data, and generates image composing. 

An image composing transmission step which transmits image composing generated 
by processing of said synthetic step to said 1st communication apparatus. 

[Claim 26]A program which controls a communications system which consists of the 

1st communication apparatus and 2nd communication apparatus, comprising: 

A demand information inputting control step by which a program which controls said 

1st communication apparatus controls an input of a user's demand information. 

A demand transmitting information control step which controls transmission to said 

2nd communication apparatus of demand information inputted by processing of said 

demand information inputting control step. 

A demand information reception control step which controls reception of said demand 
information to which a program which controls said 2nd communication apparatus is 
transmitted from said 1st communication apparatus including an image composing 



reception-control step which controls reception of image composing transmitted from 
said 2nd communication apparatus according to said demand information. 
A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to said demand information, A synthetic 
control step which controls composition of a background component image which 
consists of a background object component which constitutes background objects of 
said image data, and generation of image composing, and an image composing 
transmission-control step which controls transmission to said 1st communication 
apparatus of image composing generated by processing of said synthetic control step. 

[Claim 27]A demand information inputting control step which controls an input of a 
user's demand information to a computer which controls said 1st communication 
apparatus among computers which control a communications system which consists 
of the 1st communication apparatus and 2nd communication apparatus, A demand 
transmitting information control step which controls transmission to said 2nd 
communication apparatus of demand information inputted by processing of said 
demand information inputting control step, According to said demand information, an 
image composing reception-control step which controls reception of image composing 
transmitted from said 2nd communication apparatus is performed, A demand 
information reception control step which controls reception of said demand 
information transmitted to a computer which controls said 2nd communication 
apparatus from said 1st communication apparatus, A foreground ingredient picture 
which consists of a foreground object ingredient which constitutes a foreground 
object of image data which consists of a pixel value determined according to quantity 
of light which constitutes a picture with which it integrated in time for every pixel 
according to said demand information, Image composing generated by processing of a 
synthetic control step which controls composition of a background component image 
which consists of a background object component which constitutes background 
objects of said image data, and generation of image composing, and said synthetic 
control step, A program which performs an image composing transmission-control 
step which controls transmission to said 1st communication apparatus. 
[Claim 28]A communication apparatus comprising: 

A demand information input means which inputs a user's demand information. 

A demand information transmission means which transmits demand information 

inputted by said demand information input means to other communication apparatus. 

An image composing reception means which receives image composing transmitted 

from a communication apparatus besides the above according to said demand 

information. 



[Claim 29]A correspondence procedure comprising: 

A demand information inputting step which inputs a user's demand information. 

A demand transmitting information step which transmits demand information inputted 

by processing of said demand information inputting step to other communication 

apparatus. 

An image composing receiving step which receives image composing transmitted from 
a communication apparatus besides the above according to said demand information. 

[Claim 30]A recording medium with which a program which a computer can read is 
recorded, comprising: 

A demand information inputting control step which controls an input of a user's 
demand information. 

A demand transmitting information control step which controls transmission to other 
communication apparatus of demand information inputted by processing of said 
demand information inputting control step. 

An image composing reception-control step which controls reception of image 
composing transmitted from a communication apparatus besides the above according 
to said demand information. 

[Claim 31]A demand transmitting information control step which controls transmission 
to other communication apparatus of demand information inputted by processing of a 
demand information inputting control step which controls an input of a user's demand 
information, and said demand information inputting control step, A program which 
performs an image composing reception-control step which controls reception of 
image composing transmitted from a communication apparatus besides the above 
according to said demand information. 
[Claim 32]A communication apparatus comprising: 

A demand information receiving means which receives said demand information 
transmitted from other communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to said demand information. 
A synthesizing means which compounds a background component image which 
consists of a background object component which constitutes background objects of 
said image data, and generates said image composing. 

An image composing transmitting means which transmits said image composing 
generated by said synthesizing means to a communication apparatus besides the 
above. 



[Claim 33]A correspondence procedure comprising: 

A demand information reception step which receives said demand information 
transmitted from other communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to said demand information. 
A synthetic step which compounds a background component image which consists of 
a background object component which constitutes background objects of said image 
data, and generates said image composing. 

An image composing transmission step which transmits said image composing 
generated by processing of said synthetic step to a communication apparatus besides 
the above. 

[Claim 34]A recording medium with which a program which a computer can read is 
recorded, comprising: 

A demand information reception control step which controls reception of said demand 
information transmitted from other communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 

constitutes a foreground object of image data which consists of a pixel value 

determined according to quantity of light which constitutes a picture with which it 

integrated in time for every pixel according to said demand information. 

A synthetic control step which compounds a background component image which 

consists; of a background object component which constitutes background objects of 

said image data, and controls generation of said image composing. 

An image composing transmission-control step which controls transmission to a 

communication apparatus besides the above of said image composing generated by 

processing of said synthetic control step. 

[Claim 35]A demand information reception control step which controls reception of 
said demand information transmitted from other communication apparatus, A 
foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to said demand information, A synthetic 
control step which compounds a background component image which consists of a 
background object component which constitutes background objects of said image 
data, and controls generation of said image composing, A program which performs an 
image composing transmission-control step which controls transmission to a 



communication apparatus besides the above of said image composing generated by 
processing of said synthetic control step. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the communication apparatus and the 
method, the communications system, recording medium, and program in consideration 
of the difference from the signal and the real world which were detected by the sensor 
about a communication apparatus and a method, a communications system, a 
recording medium, and a program. 
[0002] 

[Description of the Prior Art]A sensor detects the phenomenon in the real world and 
the art of processing the sampling data which an image sensor outputs is used widely. 
[0003]For example, when objective movement speed is comparatively quick, it will 
move to the picture acquired by picturizing the object which moves in front of the 
predetermined background of standing it still with a video camera, and a Japanese 
quince will arise in it. 

[0004]The method of embedding so that it cannot recognize being enciphered by the 
flat part: as art which enciphers the above pictures, and the method of embedding 
information using the correlativity of a picture are proposed. 
[0005] 

[Problern(s) to be Solved by the Invention]When an object moves in front of a 
stationary background, mixing with the picture of the object which moves with the 
picture of not only a Japanese quince but a background by moving by mixing of the 
picture of the object which moves itself arises. Conventionally, the processing 
corresponding to the state of mixing with the picture of a background and the picture 
of the object which moves was not considered. 

[0006]This invention is made in view of such a situation, and enables it to make a 

picture encipher using the state of mixing of a picture. 

[0007] 

[Means for Solving the Problem]This invention is characterized by the 1st 
communication apparatus comprising the following. 

A demand information input means which inputs a user's demand information. 
A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 



integrated in time for every pixel according to demand information. 
A synthesizing means which compounds a background component image which 
consists of a background object component which constitutes background objects of 
image data, and generates image composing. 

An image composing output means which outputs image composing generated by 
synthesizing means. 

[0008]A user's communication apparatus can be made to output image composing to 
said image composing output means. 

[0009]In said demand information input means, with a user's demand information, a 
predetermined foreground ingredient picture, Make significant information used when 
compounding a predetermined background component image, and a predetermined 
foreground ingredient picture and a predetermined background component image input, 
and to a synthesizing means. A predetermined foreground ingredient picture inputted 
with demand information by a demand information input means and a predetermined 
background component image are made to compound based on significant information, 
and image composing can be made to generate. 

[0010]As significant information used for it when compounding a predetermined 
foreground ingredient picture, a predetermined background component image, and a 
predetermined foreground ingredient picture and a predetermined background 
component image in said demand information input means with a user's demand 
information, Make the mixture ratio of a mixing zone of a predetermined foreground 
ingredient picture and a predetermined background component image input, and to a 
synthesizing means. A predetermined foreground ingredient picture inputted with 
demand information by a demand information input means and a predetermined 
background component image are made to compound based on the mixture ratio as 
significant information, and image composing can be made to generate. 
[0011]As significant information used for it when compounding a predetermined 
foreground ingredient picture, a predetermined background component image, and a 
predetermined foreground ingredient picture and a predetermined background 
component image in said demand information input means with a user's demand 
information, Make movement quantity and the motion direction of a foreground 
ingredient picture input, and to a synthesizing means. A predetermined foreground 
ingredient picture inputted with demand information by a demand information input 
means can be moved based on movement quantity and the motion direction of as 
significant information, a Japanese quince can be adjusted, it can be made to be able 
to compound with a predetermined background component image, and image 
composing can be made to generate. 

[0012]As significant information used for it when compounding a predetermined 
foreground ingredient picture, a predetermined background component image, and a 



predetermined foreground ingredient picture and a predetermined background 
component image in said demand information input means with a user's demand 
information, Make initial position information, movement quantity, and the motion 
direction of a foreground ingredient picture input, and to a synthesizing means. Based 
on movement quantity and the motion direction of as significant information, move a 
predetermined foreground ingredient picture inputted with demand information by a 
demand information input means, adjust a Japanese quince, and the mixture ratio is 
computed, A predetermined foreground ingredient picture to which motion dotage was 
adjusted, and a predetermined background component image can be made to be able 
to compound with the computed mixture ratio based on initial position information, 
movement quantity, and the motion direction of a foreground ingredient picture of as 
significant information, and image composing can be made to generate. 
[001 3]A foreground ingredient image identifier which identifies a predetermined 
foreground ingredient picture with a user's demand information in said demand 
information input means, Make a background component image identifier which 
identifies a predetermined background component image, and significant information 
input, and to a synthesizing means. A foreground ingredient picture corresponding to a 
foreground ingredient image identifier according to a foreground ingredient image 
identifier which identifies a predetermined foreground ingredient picture with demand 
information, a background component image identifier which identifies a 
predetermined background component image, and significant information, A 
background component image corresponding to a background component image 
identifier is made to compound based on significant information, and image composing 
can be made to generate. 

[001 4]A charging means which performs accounting can be made to establish further 
according to said demand information. 

[0015]Accounting information which includes a user's identifier, an identifier of a 
communication apparatus, and amount information corresponding to demand 
information according to demand information is generated in said charging means, and 
it can be made to perform accounting to a user's financial account based on 
accounting information. 

[0016]Can make a point storage means which memorizes a point equivalent to cash 
for every user used for said accounting establish further, and to a charging means. 
Accounting information which includes a user's identifier, an identifier of a 
communication apparatus, and amount information corresponding to image composing 
according to demand information is generated, and accounting can be performed when 
only a part corresponding to amount information subtracts a point size memorized by 
point storage means. 

[0017]After accounting is completed by a charging means, a user's communication 
apparatus which accounting completed can be made to output image composing to 



said output means. 

[0018]This invention is characterized by the 1st correspondence procedure 
comprising the following. 

A demand information inputting step which inputs a user's demand information. 

A foreground ingredient picture which consists of a foreground object ingredient which 

constitutes a foreground object of image data which consists of a pixel value 

determined according to quantity of light which constitutes a picture with which it 

integrated in time for every pixel according to demand information. 

A synthetic step which compounds a background component image which consists of 

a background object component which constitutes background objects of image data, 

and generates image composing. 

An image composing output step which outputs image composing generated by 
processing of a synthetic step. 

[0019]This invention is characterized by a program of the 1st recording medium 
comprising the following. 

A demand information inputting control step which controls an input of a user's 
demand information. 

A foreground ingredient picture which consists of a foreground object ingredient which 

constitutes a foreground object of image data which consists of a pixel value 

determined according to quantity of light which constitutes a picture with which it 

integrated in time for every pixel according to demand information. 

Composition of a background component image which consists of a background object 

component which constitutes background objects of image data. 

A synthetic control step which controls generation of image composing, and an image 

composing output-control step which controls an output of image composing by which 

synthetic control step generation was carried out. 

[0020]A demand information inputting control step by which the 1st program of this 
invention controls an input of a user's demand information, A foreground ingredient 
picture which consists of a foreground object ingredient which constitutes a 
foreground object of image data which consists of a pixel value determined according 
to quantity of light which constitutes a picture with which it integrated in time for 
every pixel according to demand information, Composition of a background component 
image which consists of a background object component which constitutes 
background objects of image data, A computer is made to perform a synthetic control 
step which controls generation of image composing, and an image composing 
output-control step which controls an output of image composing by which synthetic 
control step generation was carried out. 

[0021]This invention is characterized by a communications system comprising the 



following. 

A demand information input means as which the 1st communication apparatus inputs 
a users demand information. 

A demand information transmission means which transmits demand information 
inputted by a demand information input means to the 2nd communication apparatus. 
A demand information receiving means which receives demand information which is 
equipped with an image composing reception means which receives image composing 
transmitted from the 2nd communication apparatus according to demand information, 
and to which the 2nd communication apparatus is transmitted from the 1st 
communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to demand information, A synthesizing 
means which compounds a background component image which consists of a 
background object component which constitutes background objects of image data, 
and generates image composing, and an image composing transmitting means which 
transmits image composing generated by synthesizing means to the 1 st 
communication apparatus. 

[0022]In said demand information input means, with a user's demand information, a 
predetermined foreground ingredient picture, Make significant information used when 
compounding a predetermined background component image, and a predetermined 
foreground ingredient picture and a predetermined background component image input, 
and to a synthesizing means. A predetermined foreground ingredient picture inputted 
with demand information by a demand information input means and a predetermined 
background component image are made to compound based on significant information, 
and image composing can be made to generate. 

[0023]As significant information used for it when compounding a predetermined 
foreground ingredient picture, a predetermined background component image, and a 
predetermined foreground ingredient picture and a predetermined background 
component image in said demand information input means with a user's demand 
information, Make the mixture ratio of a mixing zone of a predetermined foreground 
ingredient picture and a predetermined background component image input, and to a 
synthesizing means. A predetermined foreground ingredient picture inputted with 
demand information by a demand information input means and a predetermined 
background component image are made to compound based on the mixture ratio as 
significant information, and image composing can be made to generate. 
[0024]As significant information used for it when compounding a predetermined 
foreground ingredient picture, a predetermined background component image, and a 



predetermined foreground ingredient picture and a predetermined background 
component image in said demand information input means with a user's demand 
information, Make movement quantity and the motion direction of a foreground 
ingredient picture input, and to a synthesizing means. Based on movement quantity 
and the motion direction of as significant information, move a predetermined 
foreground ingredient picture inputted with demand information by a demand 
information input means, adjust a Japanese quince, it is made to compound with a 
foreground ingredient picture, and image composing can be made to generate. 
[0025]As significant information used for it when compounding a predetermined 
foreground ingredient picture, a predetermined background component image, and a 
predetermined foreground ingredient picture and a predetermined background 
component image in said demand information input means with a user's demand 
information, Make initial position information, movement quantity, and the motion 
direction of a foreground ingredient picture input, and a synthesizing means, Based on 
movement quantity and the motion direction of as significant information, move a 
predetermined foreground ingredient picture inputted with demand information by a 
demand information input means, adjust a Japanese quince, and the mixture ratio is 
computed, A predetermined foreground ingredient picture to which motion dotage was 
adjusted, and a predetermined background component image are made to compound 
with the computed mixture ratio based on initial position information, movement 
quantity, and the motion direction of a foreground ingredient picture of as significant 
information, and image composing can be made to generate. 

[0026]A foreground ingredient image identifier which identifies a predetermined 
foreground ingredient picture with a user's demand information in said demand 
information input means, Make a background component image identifier which 
identifies a predetermined background component image, and significant information 
input, and to a synthesizing means. A foreground ingredient picture corresponding to a 
foreground ingredient image identifier according to a foreground ingredient image 
identifier which identifies a predetermined foreground ingredient picture with demand 
information, a background component image identifier which identifies a 
predetermined background component image, and significant information, A 
background component image corresponding to a background component image 
identifier is made to compound based on significant information, and image composing 
can be made to generate. 

[0027]A charging means which performs accounting can be made to establish further 
according to said demand information. 

[0028]Accounting information which includes a user's identifier, an identifier of a 
communication apparatus, and amount information corresponding to demand 
information according to demand information is generated in said charging means, and 
it can be made to perform accounting to a user's financial account based on 



accounting information. 

[0029]Can make a point storage means which memorizes a point for every user used 
for said accounting establish further, and to a charging means. Accounting information 
which includes a user's identifier, an identifier of a communication apparatus, and 
amount information corresponding to image composing according to demand 
information is generated, and accounting can be performed when only a part 
corresponding to amount information subtracts a point size memorized by point 
storage means. 

[0030]After accounting is completed by a charging means, a user's communication 
apparatus which accounting completed can be made to output image composing to 
said output means. 

[0031]This invention is characterized by a correspondence procedure of a 
communications system comprising the following. 

A demand information inputting step into which a correspondence procedure of the 

1st communication apparatus inputs a user's demand information. 

A demand transmitting information step which transmits demand information inputted 

by processing of a demand information inputting step to the 2nd communication 

apparatus. 

A demand information reception step to which a correspondence procedure of the 2nd 
communication apparatus receives demand information transmitted from the 1st 
communication apparatus including an image composing receiving step which receives 
image composing transmitted from the 2nd communication apparatus according to 
demand information. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to demand information, A synthetic step 
which compounds a background component image which consists of a background 
object component which constitutes background objects of image data, and generates 
image composing, and an image composing transmission step which transmits image 
composing generated by processing of a synthetic step to the 1st communication 
apparatus. 

[0032]This invention is characterized by a program of the 2nd recording medium 
comprising the following. 

A demand information inputting control step by which a program which controls the 
1st communication apparatus controls an input of a user's demand information. 
A demand transmitting information control step which controls transmission to the 
2nd communication apparatus of demand information inputted by processing of a 
demand information inputting control step. 



A demand information reception control step which controls reception of demand 
information to which a program which controls the 2nd communication apparatus is 
transmitted from the 1st communication apparatus including an image composing 
reception-control step which controls reception of image composing transmitted from 
the 2nd communication apparatus according to demand information. 
A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to demand information, A synthetic control 
step which controls composition of a background component image which consists of 
a background object component which constitutes background objects of image data, 
and generation of image composing, and an image composing transmission-control 
step which controls transmission to the 1st communication apparatus of image 
composing generated by processing of a synthetic control step. 

[0033]A demand information inputting control step which controls an input of a user's 
demand information to a computer by which the 2nd program of this invention controls 
the 1st communication apparatus, A demand transmitting information control step 
which controls transmission to the 2nd communication apparatus of demand 
information inputted by processing of a demand information inputting control step, 
According to demand information, an image composing reception-control step which 
controls reception of image composing transmitted from the 2nd communication 
apparatus is performed, A demand information reception control step which controls 
reception of demand information transmitted to a computer which controls the 2nd 
communication apparatus from the 1st communication apparatus, A foreground 
ingredient picture which consists of a foreground object ingredient which constitutes 
a foreground object of image data which consists of a pixel value determined 
according to quantity of light which constitutes a picture with which it integrated in 
time for every pixel according to demand information, Composition of a background 
component image which consists of a background object component which 
constitutes background objects of image data, A synthetic control step which controls 
generation of image composing, and an image composing transmission-control step 
which controls transmission to the 1st communication apparatus of image composing 
generated by processing of a synthetic control step are performed. 
[0034]This invention is characterized by the 2nd communication apparatus comprising 
the following. 

A demand information input means which inputs a user's demand information. 
A demand information transmission means which transmits demand information 
inputted by a demand information input means to other communication apparatus. 
An image composing reception means which receives image composing transmitted 



from other communication apparatus according to demand information. 

[0035]This invention is characterized by the 2nd correspondence procedure 
comprising the following. 

A demand information inputting step which inputs a user's demand information. 
A demand transmitting information step which transmits demand information inputted 
by processing of a demand information inputting step to other communication 
apparatus. 

An image composing receiving step which receives image composing transmitted from 
other communication apparatus according to demand information. 

[0036]This invention is characterized by a program of the 3rd recording medium 
comprising the following. 

A demand information inputting control step which controls an input of a user's 
demand information. 

A demand transmitting information control step which controls transmission to other 
communication apparatus of demand information inputted by processing of a demand 
information inputting control step. 

An image composing reception-control step which controls reception of image 
composing transmitted from other communication apparatus according to demand 
information. 

[0037]A demand information inputting control step by which the 3rd program of this 
invention controls an input of a user's demand information, A demand transmitting 
information control step which controls transmission to other communication 
apparatus of demand information inputted by processing of a demand information 
inputting control step, and an image composing reception-control step which controls 
reception of image composing transmitted from other communication apparatus 
according to demand information are performed. 

[0038]This invention is characterized by the 3rd communication apparatus comprising 
the following. 

A demand information receiving means which receives demand information 
transmitted from other communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to demand information. 
A synthesizing means which compounds a background component image which 
consists of a background object component which constitutes background objects of 
image data, and generates image composing. 



An image composing transmitting means which transmits image composing generated 
by synthesizing means to other communication apparatus. 

[0039]This invention is characterized by the 3rd correspondence procedure 
comprising the following. 

A demand information reception step which receives demand information transmitted 
from other communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to demand information. 
A synthetic step which compounds a background component image which consists of 
a background object component which constitutes background objects of image data, 
and generates image composing. 

An image composing transmission step which transmits image composing generated 
by processing of a synthetic step to other communication apparatus. 

[0040]This invention is characterized by a program of the 4th recording medium 
comprising the following. 

A demand information reception control step which controls reception of demand 
information transmitted from other communication apparatus. 

A foreground ingredient picture which consists of a foreground object ingredient which 
constitutes a foreground object of image data which consists of a pixel value 
determined according to quantity of light which constitutes a picture with which it 
integrated in time for every pixel according to demand information. 
A synthetic control step which compounds a background component image which 
consists of a background object component which constitutes background objects of 
image data, and controls generation of image composing. 

An image composing transmission-control step which controls transmission to other 
communication apparatus of image composing generated by processing of a synthetic 
control step. 

[0041 ]A demand information reception control step by which the 4th program of this 
invention controls reception of demand information transmitted from other 
communication apparatus, A foreground ingredient picture which consists of a 
foreground object ingredient which constitutes a foreground object of image data 
which consists of a pixel value determined according to quantity of light which 
constitutes a picture with which it integrated in time for every pixel according to 
demand information, A synthetic control step which compounds a background 
component image which consists of a background object component which 



constitutes background objects of image data, and controls generation of image 
composing, An image composing transmission-control step which controls 
transmission to other communication apparatus of image composing generated by 
processing of a synthetic control step is performed. 

[0042]In the 1st communication apparatus of this invention, a method, and a program, 
a user's demand information being inputted and responding to demand information — 
every pixel — and, A foreground ingredient picture which consists of a foreground 
object ingredient which constitutes a foreground object of image data which consists 
of a pixel value determined according to quantity of light which constitutes a picture 
with which it integrated in time, A background component image which consists of a 
background object component which constitutes background objects of image data is 
compounded, image composing is generated, and generated image composing is 
outputted. 

[0043]A user's demand information is inputted into a communications system and a 
method of this invention by the 1st communication apparatus, It is received by image 
composing in which it is transmitted to the 2nd communication apparatus, and 
inputted demand information is transmitted from the 2nd communication apparatus 
according to demand information, and with the 2nd communication apparatus. Demand 
information transmitted from the 1st communication apparatus is received, and 
according to demand information, A foreground ingredient picture which consists of a 
foreground object ingredient which constitutes a foreground object of image data 
which consists of a pixel value determined according to quantity of light which 
constitutes a picture with which it integrated in time for every pixel, A background 
component image which consists of a background object component which 
constitutes background objects of image data is compounded, image composing is 
generated and generated image composing is transmitted to the 1st communication 
apparatus. 

[0044]In the 2nd communication apparatus of this invention, a method, and a program, 
image composing in which it is inputted, inputted demand information is transmitted to 
other communication apparatus, and a user's demand information is transmitted from 
other communication apparatus according to demand information is received. 
[0045]In the 3rd communication apparatus of this invention, a method, and a program, 
Demand information transmitted from other communication apparatus is received, and 
according to demand information, A foreground ingredient picture which consists of a 
foreground object ingredient which constitutes a foreground object of image data 
which consists of a pixel value determined according to quantity of light which 
constitutes a picture with which it integrated in time for every pixel, A background 
component image which consists of a background object component which 
constitutes background objects of image data is compounded, image composing is 
generated and generated image composing is transmitted to other communication 



apparatus. 
[0046] 

[Embodiment of the Invention] Drawing 1 is a figure showing the 1 embodiment of the 
image processing system concerning this invention. 

[0047]The image processing system of this invention for example, the network 1 top, 
such as the Internet, — the separation server 1 1 , the motion detection server 1 2, the 
field specific server 13, the mixture ratio calculation server 14, the foreground 
background separation processing server 15, and a motion — a Japanese quince — 
adjustment server 16, coding server 17, and storage-servers 18-1,18-2 and the 
synthetic server 19. the correction server 20, the purchase server 21, the sale server 
22, the retrieval server 23, the fee collection server 24, the financial server (for 
customers) 25, the financial server (for donors) 26, and the client computer 27 — and, 
The camera terminal device 28-1 thru/or 28-n have composition which is connected, 
and can deliver and receive data mutually, the separation server 11, the motion 
detection server 12, the field specific server 13, the mixture ratio calculation server 
14, the foreground background separation processing server 15, and a motion — a 
Japanese quince — the adjustment server 16, the coding server 17, the synthetic 
server 19, the correction server 20, the purchase server 21, the sale server 22, and 
the retrieval server 23. The fee collection server 24, the financial server (for 
customers) 25, and the financial server (for donors) 26, Respectively Separation 
service, motion detection service, field specific service, mixture ratio calculation 
service, Foreground-background-separation-give one's service, move and Japanese 
quince adjustment service, coding service, synthetic service, They are management or 
a server managed by the donor who provides correction service, purchase service, 
sale service, a search service, fee collection service, and financial service (the object 
for customers, and for donors). In the following explanation, when it is not necessary 
to distinguish storage-servers 18-1,18-2 and the camera terminal device 28-1 
thru/or 28-n separately, the storage servers 18 and the camera terminal device 28 
are only called. Suppose that it is the same also about other servers and apparatus. 
[0048]D rawing 2 is a figure showing the composition of the separation server 1 1 
concerning this invention. CPU(Central Processing Unit) 41 performs various kinds of 
processings according to the program memorized by ROM(Read Only Memory) 42 or 
the storage parts store 48. A program, data, etc. which CPU41 performs are suitably 
memorized by RAM(Random Access Memory) 43. These CPU41, ROM42, and RAM43 
are mutually connected by bus 44. 

[0049]The input/output interface 45 is connected to CPU41 via the bus 44 again. The 
outputting part 47 which consists of the input part 46, a display, a loudspeaker, etc. 
which consist of a keyboard, a mouse, a microphone, etc. is connected to the 
input/output interface 45. CPU41 performs various kinds of processings 
corresponding to the instructions inputted from the input part 46. And CPU41 outputs 



a picture, a sound, etc. which were obtained as a result of processing to the outputting 
part 47. 

[0050]The storage parts store 48 connected to the input/output interface 45 
comprises a hard disk etc., for example, and memorizes the program and various kinds 
of data which CPU41 performs. The communications department 49 communicates 
with an external device via the network of the Internet and others. 
[0051 ]A program may be acquired via the communications department 49 and it may 
memorize to the storage parts store 48. 

[0052]When equipped with the magnetic disk 61, the optical disc 62, the 
magneto-optical disc 63, or the semiconductor memory 64, the drive 50 connected to 
the input/output interface 45 drives them, and acquires a program, data, etc. which 
are recorded there. If needed, the program and data which were acquired are 
transmitted to the storage parts store 48, and are memorized. 

[0053]In addition, the motion detection server 12, the field specific server 13, the 
mixture ratio calculation server 14, the foreground background separation processing 
server 15, and a motion — a Japanese quince — adjustment server 16, coding server 
17, and storage-servers 18-1,18-2, the synthetic server 19, the correction server 20, 
the purchase server 21, the sale server 22, and the retrieval server 23. Since the 
basic constitution is the same as that of the separation server 1 1 about the fee 
collection server 24, the financial server (for customers) 25, the financial server (for 
donors) 26, and the client computer 27, the explanation is omitted. 
[0054] Drawing 3 is a figure showing the composition of the camera terminal device 28 
concerning this invention. The composition of the camera terminal device 28 is 
provided in the input part 76 by the sensor 76a and GPS(Global Positioning 
System)76b, and to the outputting part 77. Except that LCD(Liquid Crystal 
Display}77a is provided, it has composition of the separation server 1 1, and the same 
composition, namely, CPU71 of the camera terminal device 28, ROM72, RAM73, the 
bus 74, the input/output interface 75, the input part 76, the outputting part 77, the 
storage parts store 78, the communications department 79, the drive 80, the magnetic 
disk 91, the optical disc 92, and the magneto-optical disc 93 — and, The 
semiconductor memory 94, respectively — CPU41 of the separation server 1 1 , 
ROM42, RAM43, the bus 44, the input/output interface 45, the input part 46, the 
outputting part 47, the storage parts store 48, the communications department 49, the 
drive 50, the magnetic disk 61, the optical disc 62, and the magneto-optical disc 63 — 
and, The semiconductor memory 64 is supported, respectively. 

[0055]The sensor 76a outputs the picture which is an image sensor and was 
picturized to the input part 76. Based on the signal transmitted from the geostationary 
satellite which is not illustrated, GPS76b detects the position information (latitude and 
longitude) on the earth, and outputs the detected position information to the input 
part 76. LCD77a displays the picture outputted from an outputting part. 



[0056]Next, the separation server 1 1 is explained with reference to drawing 4 and 
drawing 5. 

[0057]As drawing 4 shows, the separation server 1 1 via the network 1 with the 
technique of mentioning later the picture inputted from the client computer 27 etc. 
separating into a foreground ingredient picture and a background component image — 
an inputted image and a foreground ingredient picture — and, It is made to output and 
accumulate in the storage servers 18 which attached and generate ID (Identifier) to 
each of a background component image, and are outputted to the client computer 27 
and which are memorized by oneself, or it outputs to the server of others on a 
network, and it is made to memorize. Here, a foreground ingredient picture shows the 
picture which has an ingredient of the motion in the picture inputted, and a 
background component image shows the picture of the stationary part which does not 
contain the motion ingredient in the picture inputted. At this time, the accounting part 
1 1a performs accounting of the fee made in separation to the fee collection server 24 
via the network 1 . As drawing 5 shows, the separation server 1 1 , If picture ID which 
specifies a picture is inputted instead of a picture, will access the retrieval server 23 
and the storage servers 18 on the network 1 which are mentioned later, or its storage 
parts store (for example, storage parts store 78 of drawing 3 ) is searched, After 
reading the image data corresponding to inputted picture ID and separating into a 
foreground ingredient picture and a background component image, ID corresponding to 
each is attached and processing corresponding to [ memorize by oneself or / output 
to the server of others on the network 1, and ] the server is performed. 
[0058]In the following explanation, although picture ID is explained as an example as 
information which specifies a picture, what is necessary is just to be the information 
which can specify a picture, and they may be image position data mentioned later. 
[0059]Next, the motion detection server 12 is explained with reference to drawing 6 
and drawing 7 . 

[0060]As drawing 6 shows, the object extraction part 12a of the motion detection 
server 12 extracts the image object in the picture inputted from the client computer 
27 etc., for example, and outputs it to the motion detection part 12b. The motion 
detection part 12b performs processing corresponding to [ memorize by oneself or / 
output to the server of others on the network 1, and ] the server which detects the 
motion vector and position information of the inputted image object, and is outputted 
to the client computer 27. Under the present circumstances, the accounting part 12c 
performs accounting of the fee made in the motion vector for every image object, and 
processing of detection of position information to the fee collection server 24 via the 
network 1. In this specification, the picture corresponding to the object in the real 
world which is the target of an image pick-up is called an object. 
[0061] As drawing 7 shows, the motion detection server 12, If picture ID which 
specifies a picture is inputted instead of a picture, will access the retrieval server 23 



and the storage servers 18 on the network 1 which are mentioned later, or its storage 
parts store (for example, storage parts store 78 of drawing 3 ) is searched, The image 
data corresponding to inputted picture ID is read, and the same processing as the 
above is performed. 

[0062]Next, the field specific server 13 is explained with reference to drawing 8 and 
drawing 9. 

[0063]As drawing 8 shows, the field specific server 13 via the network 1 using the 
object specification information that the object in the picture inputted from the client 
computer 27 etc. and its picture is specified. Each of the pixel of a picture which was 
inputted is specified as either a foreground region, a background region or a mixing 
zone, The information which shows whether it belongs to either a foreground region, a 
background region or a mixing zone for every pixel. Processing corresponding to 
[ memorize by oneself or / output to the server of others on the network 1 , and ] the 
server which generates (area information is called hereafter) and outputted to the 
client computer 27 is performed. At this time, the accounting part 13a performs 
accounting of the fee made in field specific processing to the fee collection server 24 
via the network 1. As drawing 9 shows, the field specific server 13, If picture ID which 
specifies a picture is inputted instead of a picture, will access the retrieval server 23 
and the storage servers 18 on the network 1 which are mentioned later, or its storage 
parts store (for example, storage parts store 78 of drawing 3 ) is searched, The picture 
corresponding to inputted picture ID is called, and the area information corresponding 
to the object specification information on the picture is outputted. 
[0064]Next, the mixture ratio calculation server 14 is explained with reference to 
drawing 10 and drawing 1 1 . 

[0065]As drawing 10 shows, the mixture ratio calculation server 14 via the network 1, 
For example, the picture inputted from the client computer 27 etc., the object 
specification information that the object in the picture is specified, And based on area 
information, the mixture ratio (the mixture ratio alpha is called hereafter) 
corresponding to the pixel contained in a mixing zone is computed, It outputs to the 
server of others on a network, and processing corresponding to [ memorize by oneself 
which outputs the computed mixture ratio to the client computer 27, or ] the server is 
performed. At this time, the accounting part 14a performs accounting of the fee made 
in mixture ratio calculation processing to the fee collection server 24 via the network 
1. As d rawing 1 1 shows, the mixture ratio calculation server 14, If picture ID which 
specifies a picture is inputted instead of a picture, will access the retrieval server 23 
and the storage servers 18 on the network 1 which are mentioned later, or its storage 
parts store (for example, storage parts store 78 of drawing 3 ) is searched, The picture 
corresponding to inputted picture ID is called, and the same processing as **** is 
performed. 

[0066]Next, with reference to drawing 12 and drawing 13 , the foreground background 



separation processing server 15 is explained. 

[0067]As drawing 12 shows, the foreground background separation processing server 
15 via the network 1, For example, the picture inputted from the client computer 27 
etc., the object specification information that the object in the picture is specified, 
The foreground ingredient picture which comprises only the ingredient (it is also 
hereafter called the ingredient of a foreground) of the picture corresponding to the 
object of a foreground based on area information and the mixture ratio alpha, An 
inputted image is divided into the background component image which comprises only 
the ingredient (it is also hereafter called the ingredient of a background) of a 
background, Processing corresponding to [ memorize by oneself or / output to the 
server of others on a network and ] the server which attaches ID for every picture and 
is outputted to the client computer 27 is performed. At this time, the accounting part 
15a performs accounting of the fee made in foreground background separation 
processing to the fee collection server 24 via the network 1 . As drawing 13 shows, the 
foreground background separation processing server 15, If picture ID which specifies a 
picture is inputted instead of a picture, will access the retrieval server 23 and the 
storage servers 1 8 on the network 1 which are mentioned later, or its storage parts 
store (for example, storage parts store 78 of drawing 3 ) is searched, The picture 
corresponding to inputted picture ID is called, and the same processing as **** is 
performed. 

[0068] Next, with reference to drawing 14 and drawing 15 , it moves and the Japanese 
quince adjustment server 16 is explained. 

[0069]As drawing 14 shows, the motion dotage adjustment server 16 via the network 
1, For example, the foreground ingredient picture inputted from the client computer 27 
etc., the motion included in a foreground ingredient picture based on a motion vector 
and motion blur quantity — the motion which removes a Japanese quince — the 
quantity of a Japanese quince is decreased — or a motion — the motion included in a 
foreground ingredient picture, such as making the quantity of a Japanese quince 
increase etc., — the quantity of a Japanese quince, [ adjust and ] It outputs to the 
server of others on a network, and processing corresponding to [ memorize by oneself 
which generates the foreground ingredient picture which adjusted the quantity of 
motion dotage, attaches ID for every picture, and is outputted to the client computer 
27, or ] the server is performed. At this time, the accounting part 16a performs 
accounting of the fee which moves to the fee collection server 24 via the network 1 , 
and is made in Japanese quince regulated treatment. As drawing 15 shows, move and 
the Japanese quince adjustment server 16, If foreground ingredient picture ID which 
specifies a foreground ingredient picture is inputted instead of a foreground ingredient 
picture, The retrieval server 23 and the storage servers 18 on the network 1 which are 
mentioned later are accessed, or its storage parts store (for example, storage parts 
store 78 of drawing 3 ) is searched, the foreground ingredient picture corresponding to 



inputted foreground ingredient picture ID is called, and the same processing as **** is 
performed. 

[0070]Next, the coding server 17 is explained with reference to drawing 16 and 
drawing 1 7 . 

[0071] As drawing 16 shows, the coding server 17 via the network 1, . For example, 
divide into a foreground ingredient picture and a background component image the 
picture inputted from the client computer 27 etc., give ID to each, and memorize by 
oneself. Or foreground ingredient pictures, such as URL (Universal Resource Locator) 
etc. of the server which outputs to the server of others on a network, made it 
memorize, and was made to memorize, And the foreground ingredient image position 
data which consist of numerals which show the position on the network 1 of the 
server to which the background component image was outputted (memory), and 
background component image position information are generated, and it outputs with 
the motion vector of those pictures, position information, and the information of the 
mixture ratio. The information which encoded information outputs may be any of 
encoded information, a picture, a picture, and encoded information, and can change 
the information outputted if needed. At this time, the accounting part 17a performs 
accounting of the fee made in coding processing to the fee collection server 24 via the 
network 1. As drawing 1 7 shows, the coding server 17, If picture ID which specifies a 
picture is inputted instead of a picture, will access the retrieval server 23 and the 
storage servers 18 on the network 1 which are mentioned later, or its storage parts 
store (for example, storage parts store 78 of drawing 3 ) is searched, The picture 
corresponding to inputted picture ID is called, and the same processing as **** is 
performed. 

[0072]Next, the storage servers 18 are explained with reference to drawing 18 and 
drawing 19 . 

[0073]As drawing 18 shows, it is connected on the network 1 , and the storage servers 
18 accumulate the picture transmitted from various kinds of servers, and they output 
the image position data corresponding to the stored-up picture with picture ID. By 
these image position data, the client computer 27 can be accessed via the network 1, 
and can call a desired picture, for example. That is, as drawing 19 shows, the client 
computer 27 can read the picture for which it asks by accessing the network 1 —like 
storage servers 18, and specifying picture ID corresponding to the picture considered 
as a request based on image position data. Although image position data and picture 
ID are independently explained in this specification, It may be made to make it a part 
of picture ID, and may enable it, as for image position data, to recognize which server 
on the network 1 things can be carried out and memorizes [ which recognizes image 
position data by picture ID in this case ] (accumulation or processing). It may be made 
for the storage servers 18 to memorize a motion vector, position information, the 
mixture ratio, and motion blur quantity besides image data. 



[0074]Next, the synthetic server 19 is explained with reference to drawing 20 and 
drawing 21 . 

[0075]As drawing 20 shows, the synthetic server 19 via the network 1 f For example, 
two pictures of the pictures A and B inputted from the client computer 27 etc., 
Combine the pictures A and B from a motion vector and position information, the 
mixture ratio, and motion blur quantity, and image composing (A+B) is generated, 
Processing corresponding to [ memorize by oneself or / output to the server of 
others on the network 1, and ] the server outputted to the client computer 27 is 
performed. In this case, as for the pictures A and B, both will be compounded by 
treating one side as a foreground ingredient picture, and treating another side as a 
background component image. At this time, the accounting part 19a performs 
accounting of the fee made in a compositing process to the fee collection server 24 
via the network 1. As drawing 21 shows, the synthetic server 19, If picture A-ID and 
B-ID which specify a picture are inputted instead of the pictures A and B, The 
retrieval server 23 and the storage servers 18 on the network 1 which are mentioned 
later are accessed, or its storage parts store (for example, storage parts store 78 of 
drawing 3) is searched, the picture corresponding to picture A-ID and B-ID which 
were inputted is called, and the same processing as **** is performed. 
[0076] Next, the correction server 20 is explained with reference to drawing 22 and 
drawing 23 . 

[0077]As drawing 22 shows, the correction server 20 via the network 1, For example, 
the picture inputted from the client computer 27 etc., Processing corresponding to 
[ memorize by oneself or / output to the server of others on a network and ] the 
server which corrects based on a motion vector and position information, the mixture 
ratio, and motion blur quantity, generates a corrected image, and is outputted to the 
client computer 27 is performed. At this time, the accounting part 20a performs 
accounting of the fee made in correction processing to the fee collection server 24 via 
the network 1. As drawing 23 shows, the correction server 20, If picture ID which 
specifies a picture is inputted instead of a picture, will access the retrieval server 23 
and the storage servers 18 on the network 1 which are mentioned later, or its storage 
parts store (for example, storage parts store 78 of drawing 3 ) is searched, The picture 
corresponding to inputted picture ID is called, and the same processing as **** is 
performed. 

[0078]Next, the purchase server 21 is explained with reference to drawing 24 . 
[0079]As drawing 24 shows, the purchase server 21 via the network 1, For example, if 
picture ID which specifies the picture which the client computer 27 etc. are operated 
by the user who wishes the purchase of a predetermined picture, and wishes to 
purchase is inputted, A corresponding picture is accessed at the separation server 1 1 , 
the storage servers 18, the synthetic server 19, or the correction server 20 on the 
network 1, etc., a corresponding picture is called, and it outputs to the client computer 



27. At this time, the accounting part 21a performs accounting of the fee concerning 
the picture purchased to the fee collection server 24 via the network 1. 
[0080]Next, the sale server 22 is explained with reference to drawing 25 . 
[0081 ]As drawing 25 shows, the sale server 22 via the network 1 For example, the 
separation server 11, the synthetic server 19, Or the separation picture, image 
composing which were generated by the correction server 20 and which a 
predetermined user owns, When the picture which wishes sale of a corrected image is 
inputted, the picture which wishes to sell off Or the separation server 11 on the 
network 1, the storage servers 18, the synthetic server 19, Or make the correction 
server 20 etc. memorize and the accounting part 22a receives the fee collection 
server 24 via the network 1 , Accounting of the fee concerning the picture to sell off is 
performed (in this case, the donor of sale processing service will perform payment 
processing for the price corresponding to the picture sold off to the user who wishes 
to sell off). 

[0082]Next, the retrieval server 26 is explained with reference to drawing 26 . 
[0083]The information the retrieval server 26 indicates the feature of the picture 
which a user considers as a request with the client computer 1 to be, for example, The 
picture picturized now or the picture picturized beforehand is searched with the 
camera terminal device 28-1 on the network 1 thru/or 28-n from the physical position 
information on the camera terminal device 28-1 thru/or 28-n, and it outputs to the 
client computer 1 as a demand picture. At this time, the accounting part 23a performs 
accounting of the fee made in retrieval processing to the fee collection server 24 via 
the network 1. 

[0084]In this specification, with coding, it shall call it coding to change into the 
foreground ingredient picture acquired based on image data, a background component 
image, a motion vector, position information, motion blur quantity, and the information 
on the mixture ratio, and the data shall be called coding data. 
[0085]D rawing 27 is a block diagram showing the separation server 11. 
[0086]It is not asked whether each function of the separation server 1 1 is realized by 
hardware, or software realizes. That is, even if it considers each block diagram of this 
specification to be a block diagram of hardware, it may be considered to be a 
functional block diagram by software. 

[0087]The inputted image supplied to the separation server 1 1 is supplied to the 
object extraction part 101, the field specific part 103, the mixture ratio calculation 
part 104, and the foreground background separation part 105. 

[0088]The object extraction part 101 extracts coarsely the image object 
corresponding to the object of the foreground included in an inputted image, and 
supplies the extracted image object to the motion detection part 102. The object 
extraction part 101 is, detecting the outline of the image object corresponding to the 
object of the foreground included in an inputted image for example, and extracts the 



image object corresponding to the object of a foreground coarsely. 
[0089]The object extraction part 101 extracts coarsely the image object 
corresponding to the object of the background included in an inputted image, and 
supplies the extracted image object to the motion detection part 102. The object 
extraction part 101 extracts the image object corresponding to the object of a 
background from the difference of an inputted image and the image object 
corresponding to the object of the extracted foreground coarsely, for example. 
[0090] For example, the object extraction part 101, It may be made to extract coarsely 
the image object corresponding to the object of a foreground, and the image object 
corresponding to the object of a background from the difference of the picture of the 
background memorized by the background memory provided in the inside, and an 
inputted image. 

[0091]The motion detection part 102, for example with techniques, such as the block 
matching method, a gradient method, a phase correlation technique, and the pel 
recursive method. The motion vector of the image object corresponding to the object 
of the foreground extracted coarsely is computed, and the position information on the 
computed motion vector and a motion vector (information which pinpoints the position 
of the pixel corresponding to a motion vector) is supplied to the field specific part 103 
and the motion dotage controller 106. 

[0092]The information corresponding to movement quantity v is included in the 
motion vector which the motion detection part 102 outputs. 

[0093]The motion detection part 102 moves and it may be made to, output the motion 
vector for every image object to the Japanese quince controller 106 for example, with 
the picture element position information which specifies a pixel as an image object. 
[0094] Movement quantity v is a value which expresses a pixel interval for change of 
the position of the picture corresponding to the object which is moving as a unit. For 
example, while moving so that it may be displayed on the position which the picture of 
the object corresponding to a foreground left by 4 pixels in the following frame on the 
basis of a certain frame, movement quantity v of the picture of the object 
corresponding to a foreground is set to 4. 

[0095]The object extraction part 101 and the motion detection part 102 are needed 
when adjusting the motion blur quantity corresponding to the object which is moving. 
[0096]The field specific part 103 each of the pixel of a picture which was inputted A 
foreground region, It specifies as either a background region or a mixing zone, and the 
information which shows whether it belongs to either a foreground region, a 
background region or a mixing zone for every pixel is supplied to the mixture ratio 
calculation part 104, the foreground background separation part 105, and the motion 
dotage controller 106. 

[0097]Based on an inputted image and the area information supplied from the field 
specific part 103, the mixture ratio calculation part 104 computes the mixture ratio 



corresponding to the pixel contained in a mixing zone, and supplies the computed 
mixture ratio to the foreground background separation part 105. 

[0098]the mixture ratio — alpha is a value which shows the rate of the ingredient (it is 
also hereafter called the ingredient of a background) of the picture corresponding to 
the object of a background in a pixel value, as shown in the formula (3) mentioned later. 
[0099]The area information to which the foreground background separation part 105 
was supplied from the field specific part 103, and the mixture ratio supplied from the 
mixture ratio calculation part 104 — with the foreground ingredient picture which 
comprises only the ingredient (it is also hereafter called the ingredient of a 
foreground) of the picture corresponding to the object of a foreground based on alpha. 
An inputted image is divided into the background component image which comprises 
only the ingredient of a background, it moves and a foreground ingredient picture is 
supplied to the Japanese quince controller 106 and the selecting part 107. 
Considering the separated foreground ingredient picture as a final output is also 
considered. Only a foreground and a background can be specified and an exact 
foreground and background can be acquired compared with the separated method 
without taking the conventional mixing zone into consideration. 

[0100]The motion dotage controller 106 determines the batch which shows one or 
more pixels contained in a foreground ingredient picture based on movement quantity 
v and area information which a motion vector shows. A batch is data which specifies 
the pixel of one group which is the target of processing of adjustment of the quantity 
of motion dotage. 

[0101]The motion dotage controller 106 was inputted into the separation server 11, 
move, and The amount of Japanese quince adjustments, The motion vector supplied 
from the foreground ingredient picture supplied from the foreground background 
separation part 105, and the motion detection part 102, and its position information, 
and the motion included in a foreground ingredient picture based on a batch — the 
motion which removes a Japanese quince — the quantity of a Japanese quince is 
decreased — or a motion — the motion included in a foreground ingredient picture, 
such as making the quantity of a Japanese quince increase etc., — adjusting the 
quantity of a Japanese quince — a motion — the foreground ingredient picture which 
adjusted the quantity of the Japanese quince is outputted to the selecting part 107. A 
motion vector and its position information may not be used. 

[0102]Here, motion dotage means distortion included in the picture corresponding to 
the object which is moving produced with the motion of the object in the real world 
which is the target of an image pick-up, and the characteristic of an image pick-up of 
a sensor. 

[0103]The selecting part 107 chooses either one of the foreground ingredient picture 
supplied from the foreground background separation part 105 based on the selection 
signal corresponding to the user's selection, and the foreground ingredient picture to 



which was supplied from the motion dotage controller 106, it moved, and the quantity 
of the Japanese quince was adjusted, for example, and outputs the selected 
foreground ingredient picture. 

[0104]Next, with reference to drawing 28 thru/or drawing 43 , the inputted image 
supplied to the separation server 1 1 is explained. 

[0105]D rawing 28 is a figure explaining the image pick-up by the sensor 76a. The 
sensor 76a comprises a CCD video camera etc. which were provided with the CCD 
(Charge-Coupled Device) area sensor which is a solid state image pickup device, for 
example. The object corresponding to a foreground in the real world moves at a level 
with right-hand side from the left-hand side in a figure between the object 
corresponding to a background in the real world, and sensors, for example. 
[0106]The sensor 76a picturizes the object corresponding to a foreground with the 
object corresponding to a background. The sensor 76a outputs the picturized picture 
per frame. For example, the sensor 76a outputs the picture which comprises 30 
frames in 1 second. Exposure time of the sensor 76a can be made into 1 / 30 seconds. 
Exposure time is a period until it ends conversion to the electric charge of the 
inputted light, after starting conversion to the electric charge of the light into which 
the sensor 76a was inputted. Hereafter, exposure time is also called shutter time. 
[01 07]D rawing 29 is a figure explaining arrangement of a pixel. In drawing 29 , A thru/ or 
I show each pixel. The pixel is arranged on the flat surface corresponding to a picture. 
One sensing element corresponding to one pixel is arranged on the sensor 76a. When 
the sensor 76a picturizes a picture, one sensing element outputs the pixel value 
corresponding to one pixel which constitutes a picture. For example, the position of 
the direction of X of a sensing element is equivalent to the position of the transverse 
direction on a picture, and the position of the direction of Y of a sensing element is 
equivalent to the position of the lengthwise direction on a picture. 
[0108]As shown in drawing 30 , the sensing element which is CCD changes the period 
corresponding to shutter time, and the inputted light into an electric charge, and 
accumulates the changed electric charge. The quantity of an electric charge is 
proportional to the inputted intensity of light and the time when light is inputted 
mostly. In the period corresponding to shutter time, the sensing element adds the 
electric charge changed from the inputted light to the already accumulated electric 
charge. That is, a sensing element integrates with the period corresponding to shutter 
time, and the light inputted, and accumulates the electric charge of the quantity 
corresponding to the light with which it integrated. It can be said that a sensing 
element has the storage effect to time. 

[0109]The electric charge accumulated in the sensing element is transformed into a 
pressure value by the circuit which is not illustrated, and a pressure value is further 
changed and outputted to the pixel value of digital data etc. Therefore, each pixel 
value outputted from the sensor 76a has the value projected on the one-dimensional 



space which is the result of integrating with a certain portion into which the object 
corresponding to a foreground or a background has breadth spatially about shutter 
time. 

[01 10]The separation server 1 1 extracts the significant information alpha buried in the 
output signal, for example, the mixture ratio, by operation of such accumulation of the 
sensor 76a. The separation server 1 1 adjusts the quantity which is distortion by the 
image object of a foreground itself being mixed to produce, for example, the quantity 
of motion dotage, etc. The separation server 1 1 adjusts the quantity of distortion 
produced when the image object of a foreground and the image object of a background 
are mixed. 

[01 1 1] Drawing 31 is a figure explaining the picture acquired by picturizing the object 
corresponding to the foreground currently moved, and the object corresponding to a 
stationary background. Drawing 31 (A) shows the picture acquired by picturizing the 
object corresponding to the foreground accompanied by a motion, and the object 
corresponding to a stationary background. In the example shown in drawing 31 (A), the 
object corresponding to a foreground is moving to the right from the left horizontally 
to the screen. 

[01 1 2] Drawing 31 (B) is the model figure which developed the pixel value 
corresponding to one line of the picture shown in drawing 31 (A) to the time direction. 
The transverse direction of drawing 31 (B) corresponds in the direction X of space of 
drawing 31 (A) . 

[01 1 3]The pixel value comprises only an ingredient of the picture corresponding to the 
ingredient of a background, i.e., the object of a background, in the pixel of a 
background region. The pixel value comprises only an ingredient of the picture 
corresponding to the ingredient of a foreground, i.e., the object of a foreground, in the 
pixel of a foreground region. 

[01 14]As for the pixel of the mixing zone, the pixel value comprises an ingredient of a 
background, and an ingredient of the foreground. Since the pixel value comprises an 
ingredient of a background, and an ingredient of the foreground, the mixing zone can 
say it also as a distortion area. A mixing zone is further classified into a covered 
background region and an uncovered background region. 

[01 15]A covered background region is a mixing zone of the position corresponding to 
the front end part of the direction of movement of the object of a foreground to a 
foreground region, and the field where a background component is covered by the 
foreground corresponding to the passage of time is said. 

[0116]On the other hand, an uncovered background region is a mixing zone of the 
position corresponding to the rear end part of the direction of movement of the object 
of a foreground to a foreground region, and the field where a background component 
appears corresponding to the passage of time is said. 

[0117]Thus, a picture including a foreground region, a background region, a covered 



background region, or an uncovered background region is inputted into the field 
specific part 103, the mixture ratio calculation part 104, and the foreground 
background separation part 105 as an inputted image. 

[01 18]D rawing 32 is a figure explaining above background regions and foreground 
regions, a mixing zone, a covered background region, and an uncovered background 
region. When it corresponds to the picture shown in drawing 31 , a background region is 
a stationary part, a foreground region is a motion portion, the covered background 
region of a mixing zone is a portion which changes from a background to a foreground, 
and the uncovered background region of a mixing zone is a portion which changes 
from a foreground to a background. 

[01 19]D rawing 33 is the model figure in the picture which picturized the object 
corresponding to a stationary foreground, and the object corresponding to a 
stationary background which developed the pixel value of the pixel adjacently located 
in a line with one row to the time direction. For example, the pixel located in a line on 
one line of a screen can be chosen as a pixel adjacently located in a line with one row. 
[0120]The pixel value of F01 thru/or F04 which are shown in drawing 33 is a pixel 
value of the pixel corresponding to the stationary object of a foreground. The pixel 
value of B01 thru/or B04 which are shown in drawing 33 is a pixel value of the pixel 
corresponding to the stationary object of a background. 

[0121]The lengthwise direction in drawing 33 corresponds to time, and time passes 
toward the bottom since on in a figure. The position of the top chord of the rectangle 
in drawing 33 corresponds to the time which starts conversion to the electric charge 
of the light into which the sensor 76a was inputted, and the position of the lower side 
of the rectangle in drawing 33 corresponds to the time which ends conversion to the 
electric charge of the light into which the sensor 76a was inputted. That is, the 
distance from the top chord of the rectangle in drawing 33 t o the lower side 
corresponds to shutter time. 

[0122]Below, the case where shutter time and a frame interval are the same is 
explained at an example. 

[0123]The transverse direction in drawing 33 corresponds in the direction X of space 
explained by drawing 31 . More specifically in the example shown in drawing 33 , the 
distance to the right-hand side of the rectangle indicated to be "B04" from the left 
side of the rectangle indicated to be "F01" in drawing 33 corresponds by 8 times, i.e., 
the continuous interval of eight pixels, the pitch of a pixel. 

[0124]When the object of a foreground and the object of a background are standing it 
still, in the period corresponding to shutter time, the light inputted into the sensor 76a 
does not change. 

[01 25]Here, the period corresponding to shutter time is divided during the two or more 
same length. For example, if the virtual number of partitions is set to 4, the model 
figure shown in drawing 33 can be expressed as a model shown in drawing 9 . The 



virtual number of partitions is set up corresponding to movement quantity [ within the 
shutter time of the object corresponding to a foreground ] v etc. For example, 
corresponding to movement quantity v which is 4, the virtual number of partitions is 
set to 4, and the period corresponding to shutter time is divided into four. 
[0126]The line of the top in a figure corresponds to the period when the shutter 
opened in and the beginning was divided. A shutter opens the 2nd line from on in a 
figure, and it corresponds to the 2nd divided period. A shutter opens the 3rd line from 
on in a figure, and it corresponds to the 3rd divided period. A shutter opens the 4th 
line from on in a figure, and it corresponds to the 4th divided period. 
[0127]Hereafter, shutter time/v calls the shutter time divided corresponding to 
movement quantity v. 

[0128]Since the light inputted into the sensor 76a does not change while the object 
corresponding to a foreground is standing it still, the ingredients F01/v of a foreground 
are equal to the value which **(ed) the pixel value F01 by the virtual number of 
partitions. Similarly while the object corresponding to a foreground is standing it still, 
the ingredients F02/v of a foreground, It is equal to the value which **(ed) the pixel 
value F02 by the virtual number of partitions, and the ingredients F03/v of a 
foreground are equal to the value which **(ed) the pixel value F03 by the virtual 
number of partitions, and their ingredients F04/v of a foreground are equal to the 
value which **(ed) the pixel value F04 by the virtual number of partitions. 
[0129]Since the light inputted into the sensor 76a does not change while the object 
corresponding to a background is standing it still, the ingredients B01 /v of a 
background are equal to the value which **(ed) the pixel value B01 by the virtual 
number of partitions. Similarly, while the object corresponding to a background is 
standing it still, the ingredients B02/v of a background are equal to the value which 
**(ed) the pixel value B02 by the virtual number of partitions, and that of B03/v are 
equal to the value which **(ed) the pixel value B03 by the virtual number of partitions, 
and B04/their v is equal to the value which **(ed) the pixel value B04 by the virtual 
number of partitions. 

[0130]Namely, since the light corresponding to the object of a foreground inputted 
into the sensor 76a does not change in the period corresponding to shutter time when 
the object corresponding to a foreground is standing it still, The ingredients F01/v of 
the foreground corresponding to [ a shutter opens and ] the first shutter time/v, A 
shutter opens and the ingredients F01/v of the foreground corresponding to the 2nd 
shutter time / v, the ingredients F01/v of the foreground corresponding to [ a shutter 
opens and ] the 3rd shutter time / v, and the ingredients F01/v of the foreground 
corresponding to [ a shutter opens and ] the 4th shutter time / v serve as the same 
value. F02/v thru/or F04/v also have the same relation as F01/v. 
[0131]Since the light corresponding to the object of a background inputted into the 
sensor 76a does not change in the period corresponding to shutter time when the 



object corresponding to a background is standing it still, The ingredients B01/v of the 
background corresponding to [ a shutter opens and ] the first shutter time/v, A 
shutter opens and the ingredients B01/v of the background corresponding to the 2nd 
shutter time / v f the ingredients B01/v of the background corresponding to [ a shutter 
opens and ] the 3rd shutter time / v, and the ingredients B01/v of the background 
corresponding to [ a shutter opens and ] the 4th shutter time / v serve as the same 
value. B02/v thru/or B04/v also have the same relation. 

[0132]Next, the object corresponding to a foreground moves and the case where the 
object corresponding to a background is standing it still is explained. 
[0133]D rawing 35 is the model figure which developed the pixel value of the pixel on 
one line to the time direction, including a covered background region in case the 
object corresponding to a foreground moves toward the right-hand side in a figure. In 
drawing 35 , movement quantity v of a foreground is 4. Since one frame is short time, 
the object corresponding to a foreground is a rigid body, and it can be assumed that it 
is moving at uniform velocity. In drawing 35 , the picture of the object corresponding to 
a foreground moves so that it may be displayed on right-hand side by 4 pixels in the 
following frame on the basis of a certain frame. 

[0134]In drawing 35 , the 4th pixel belongs to a foreground region from the pixel 
thru/or the left of most left-hand side. In drawing 35 , the left to the 5th thru/or the 
7th pixel belong to the mixing zone which is a covered background region from the left. 
I n drawing 35 , the pixel of most right-hand side belongs to a background region. 
[0135]Since it is moving so that the object corresponding to a foreground may cover 
the object corresponding to a background with the passage of time, when the 
ingredient contained in the pixel value of the pixel belonging to a covered background 
region has a period corresponding to shutter time, it is replaced with the ingredient of 
a foreground from the ingredient of a background. 

[01 36]For example, the pixel value M which attached the thick line frame in drawing 35 

is expressed with a formula (1). 

[0137] 

M=B02/v+B02/v+F07/v+F06/v (1) 

[0138]For example, since the 5th pixel contains the ingredient of the foreground 
corresponding to three shutter time / v including the ingredient of the background 
corresponding to one shutter time / v from the left, the mixture ratio alpha of the 5th 
pixel is 1/4 from the left. Since the 6th pixel contains the ingredient of the foreground 
corresponding to two shutter time / v including the ingredient of the background 
corresponding to two shutter time / v from the left, the mixture ratio alpha of the 6th 
pixel is 1/2 from the left. Since the 7th pixel contains the ingredient of the foreground 
corresponding to one shutter time / v including the ingredient of the background 
corresponding to three shutter time / v from the left, the mixture ratio alpha of the 
7th pixel is 3/4 from the left. 



[0139]Since it can assume that it moves at uniform velocity so that the object 
corresponding to a foreground may be a rigid body and the picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, For example, the 
shutter of the 4th pixel opens from the left in drawing 35 , the shutter of the 5th pixel 
opens the first ingredients F07/v of the foreground of shutter time/v from the left in 
drawing 35 , and it is equal to the ingredient of the foreground corresponding to the 
2nd shutter time / v. Similarly, the shutter of the 6th pixel opens the ingredients 
F07/v of a foreground from the left in drawing 35 , and the shutter of the 7th pixel 
opens them from the ingredient of the foreground corresponding to the 3rd shutter 
time / v, and the left in drawing 35 , and they are equal to the ingredient of the 
foreground corresponding to the 4th shutter time / v respectively. 
[0140]Since it can assume that it moves at uniform velocity so that the object 
corresponding to a foreground may be a rigid body and the picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, For example, the 
shutter of the 3rd pixel opens from the left in drawing 35 , the shutter of the 4th pixel 
opens the ingredients F06/v of the foreground of the first shutter time/v from the left 
in drawing 35 , and it is equal to the ingredient of the foreground corresponding to the 
2nd shutter time / v. Similarly, the shutter of the 5th pixel opens the ingredients 
F06/v of a foreground from the left in drawing 35 , and the shutter of the 6th pixel 
opens them from the ingredient of the foreground corresponding to the 3rd shutter 
time / v, and the left in drawing 35 , and they are equal to the ingredient of the 
foreground corresponding to the 4th shutter time / v respectively. 
[0141]Since it can assume that it moves at uniform velocity so that the object 
corresponding to a foreground may be a rigid body and the picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, For example, the 
shutter of the 2nd pixel opens from the left in drawing 35 , the shutter of the 3rd pixel 
opens the ingredients F05/v of the foreground of the first shutter time/v from the left 
in drawing 35 , and it is equal to the ingredient of the foreground corresponding to the 
2nd shutter time / v. Similarly, the shutter of the 4th pixel opens the ingredients 
F05/v of a foreground from the left in drawing 35 , and the shutter of the 5th pixel 
opens them from the ingredient of the foreground corresponding to the 3rd shutter 
time / v, and the left in drawing 35 , and they are equal to the ingredient of the 
foreground corresponding to the 4th shutter time / v respectively. 
[0142]Since it can assume that it moves at uniform velocity so that the object 
corresponding to a foreground may be a rigid body and the picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, For example, the 
shutter of the pixel by the side of the leftmost in drawing 35 opens, the shutter of the 
2nd pixel opens the ingredients F04/v of the foreground of the first shutter time/v 
from the left in drawing 35 , and it is equal to the ingredient of the foreground 
corresponding to the 2nd shutter time / v. Similarly, the shutter of the 3rd pixel opens 



the ingredients F04/v of a foreground from the left in drawing 35 , and the shutter of 
the 4th pixel opens them from the ingredient of the foreground corresponding to the 
3rd shutter time / v, and the left in drawing 35 , and they are equal to the ingredient of 
the foreground corresponding to the 4th shutter time / v respectively. 
[0143]Since it moves in this way and the field of the foreground corresponding to the 
object which is moving contains a Japanese quince, it can also be said to be a 
distortion area. 

[0144]D rawing 36 is the model figure which developed the pixel value of the pixel on 
one line to the time direction, including an uncovered background region in case a 
foreground moves toward the right-hand side in a figure. In drawing 36 , movement 
quantity v of a foreground is 4. Since one frame is short time, the object 
corresponding to a foreground is a rigid body, and it can be assumed that it is moving 
at uniform velocity. In drawing 36 , the picture of the object corresponding to a 
foreground moves to right-hand side by 4 pixels in the following frame on the basis of 
a certain frame. 

[0145]In drawing 36 , the 4th pixel belongs to a background region from the pixel 
thru/or the left of most left-hand side. In drawing 36 , the left to the 5th thru/or the 
7th pixel belong to the mixing zone which is an uncovered background from the left. In 
drawing 36 , the pixel of most right-hand side belongs to a foreground region. 
[0146]Since it is moving so that the object corresponding to the foreground which had 
covered the object corresponding to a background may be removed from before the 
object corresponding to a background with the passage of time, When the ingredient 
contained in the pixel value of the pixel belonging to an uncovered background region 
has a period corresponding to shutter time, it is replaced with the ingredient of a 
background from the ingredient of a foreground. 

[0147]For example, pixel value M' which attached the thick line frame in drawing 36 is 

expressed with a formula (2). 

[0148] 

M'=F02/v+F01/v+B26/v+B26/v (2) 

[01 49] For example, since the 5th pixel contains the ingredient of the foreground 
corresponding to one shutter time / v including the ingredient of the background 
corresponding to three shutter time / v from the left, the mixture ratio alpha of the 
5th pixel is 3/4 from the left. Since the 6th pixel contains the ingredient of the 
foreground corresponding to two shutter time / v including the ingredient of the 
background corresponding to two shutter time / v from the left, the mixture ratio 
alpha of the 6th pixel is 1/2 from the left. Since the 7th pixel contains the ingredient 
of the foreground corresponding to three shutter time / v including the ingredient of 
the background corresponding to one shutter time / v from the left, the mixture ratio 
alpha of the 7th pixel is 1/4 from the left. 

[0150]When a formula (1) and a formula (2) are generalized more, the pixel value M is 



expressed with a formula (3). 

[0151] 

[Equation 1] 

M=a-B + ^Fi/v (3) 



Here, alpha is the mixture ratio. B is a pixel value of a background and Fi/v is an 
ingredient of a foreground. 

[0152]The object corresponding to a foreground is a rigid body, and can assume that it 
moves at uniform velocity, and since movement quantity v is 4, For example, the 
shutter of the 5th pixel opens from the left in drawing 36 , the shutter of the 6th pixel 
opens the first ingredients F01/v of the foreground of shutter time/v from the left in 
drawing 36 , and it is equal to the ingredient of the foreground corresponding to the 
2nd shutter time / v. Similarly, the shutter of the 7th pixel opens F01/v from the left 
in drawing 36 , and the shutter of the 8th pixel opens it from the ingredient of the 
foreground corresponding to the 3rd shutter time / v, and the left in drawing 36 , and it 
is equal to the ingredient of the foreground corresponding to the 4th shutter time / v 
respectively. 

[0153]An object corresponding to a foreground is a rigid body, and can assume that it 
moves at uniform velocity, and since the virtual number of partitions is 4, For example, 
a shutter of the 6th pixel opens from the left in drawing 36 , a shutter of the 7th pixel 
opens the first ingredients F02/v of a foreground of shutter time/v from the left in 
drawing 36 , and it is equal to an ingredient of a foreground corresponding to the 2nd 
shutter time / v. Similarly, a shutter of the 8th pixel opens the ingredients F02/v of a 
foreground from the left in drawing 36 , and they are equal to an ingredient of a 
foreground corresponding to the 3rd shutter time / v. 

[0154]An object corresponding to a foreground is a rigid body, and can assume that it 
moves at uniform velocity, and since movement quantity v is 4, For example, a shutter 
of the 7th pixel opens from the left in drawing 36 , a shutter of the 8th pixel opens the 
first ingredients F03/v of a foreground of shutter time/v from the left in drawing 36 , 
and it is equal to an ingredient of a foreground corresponding to the 2nd shutter time 
/v. 

[0155]In explanation of drawing 34 thru/or drawing 36 , although it was explained that 
the virtual number of partitions was 4, the virtual number of partitions corresponds to 
movement quantity v. Generally movement quantity v corresponds to movement 
speed of an object corresponding to a foreground. For example, while moving so that 
an object corresponding to a foreground may be displayed on right-hand side by 4 
pixels in the following frame on the basis of a certain frame, movement quantity v is 
set to 4. Corresponding to movement quantity v, the virtual number of partitions is set 
to 4. While similarly moving, for example so that an object corresponding to a 



foreground may be displayed on left-hand side by 6 pixels in the following frame on the 
basis of a certain frame, movement quantity v is set to 6 and the virtual number of 
partitions is set to 6. 

[01 56]Relation between a mixing zone which comprises a foreground region, a 
background region and a covered background region which were explained to drawing 
37 and drawing 38 above, or an uncovered background region, and an ingredient of a 
foreground corresponding to divided shutter time and an ingredient of a background is 
shown. 

[0157] Drawing 37 shows an example which extracted a pixel of a foreground region, a 
background region, and a mixing zone from a picture including a foreground 
corresponding to an object which is moving in a stationary background front. In an 
example shown in drawing 37 , an object corresponding to a foreground is moving 
horizontally to a screen. 

[0158]Frame #n+1 is the next frame of frame #n, and frame #n+2 is the next frame of 
frame #n+1 . 

[0159]A pixel of a foreground region, a background region, and a mixing zone which 
were extracted from either of frame #n thru/or frame #n+2 is extracted, movement 
quantity v is set to 4, and a model which developed a pixel value of an extracted pixel 
to a time direction is shown in drawing 38 . 

[0160]Since an object corresponding to a foreground moves, a pixel value of a 
foreground region comprises an ingredient of four different foregrounds corresponding 
to a period of shutter time/v. For example, a pixel located most in left-hand side 
among oixels of a foreground region shown in drawing 38 comprises F01/v, F02/v, 
F03/v, and F04/v. That is, a pixel of a foreground region moves and contains a 
Japanese quince. 

[0161]Since an object corresponding to a background is standing it still, in a period 
corresponding to shutter time, light corresponding to a background inputted into the 
sensor 76a does not change. In this case, a pixel value of a background region moves 
and does not contain a Japanese quince. 

[0162]A pixel value of a pixel belonging to a mixing zone which comprises a covered 
background region or an uncovered background region comprises an ingredient of a 
foreground, and an ingredient of a background. 

[0163]Next, while a picture corresponding to an object is moving, it is the pixel in two 
or more frames adjacently located in a line with one row, and a model which developed 
a pixel value of a pixel of the same position to a time direction on a frame is explained. 
For example, while a picture corresponding to an object is moving horizontally to a 
screen, a pixel located in a line on one line of a screen can be chosen as a pixel 
adjacently located in a line with one row. 

[0164] Drawing 39 is a pixel which three frames of a picture which picturized an object 
corresponding to a stationary background adjoined, and has been located in a line with 



one row, and is the model figure which developed a pixel value of a pixel of the same 
position to a time direction on a frame. Frame #n is the next frame of frame #n-1, and 
frame #n+1 is the next frame of frame #n. Other frames are called similarly. 
[0165]A pixel value of B01 thru/or B12 which are shown in drawing 39 is a pixel value 
of a pixel corresponding to a stationary object of a background. Since an object 
corresponding to a background is standing it still, a pixel value of a corresponding pixel 
does not change in frame #n-1 thru/or the frame n+1. For example, a pixel in frame #n 
corresponding to a position of a pixel which has a pixel value of B05 in frame #n~1 , and 
a pixel in frame #n+1 have a pixel value of B05, respectively. 

[01 66] Drawing 40 is a pixel which three frames of a picture which picturized an object 
corresponding to a foreground moved to right-hand side in a figure adjoined, and has 
been located in a line with one row with an object corresponding to a stationary 
background, and is the model figure which developed a pixel value of a pixel of the 
same position to a time direction on a frame. A model shown in drawing 40 includes a 
covered background region. 

[0167]In drawing 40 , since it moves so that an object corresponding to a foreground 
may be a rigid body, and can assume that it moves at uniform velocity and a picture of 
a foreground may be displayed on 4-pixel right-hand side in the following frame, 
movement quantity v of a foreground is 4 and the virtual number of partitions is 4. 
[0168]For example, a shutter of a pixel by the side of the leftmost of frame #n-1 in 
drawing 40 opens, an ingredient of a foreground of the first shutter time/v is set to 
F12/v, a shutter of the 2nd pixel opens it from the left in drawing 40 , and an ingredient 
of the 2nd foreground of shutter time/v is also set to F1 2/v. A shutter of the 3rd pixel 
opens from the left in drawing 40 , a shutter of the 4th pixel opens from an ingredient 
of the 3rd foreground of shutter time/v, and the left in drawing 40 , and an ingredient of 
the 4th foreground of shutter time/v is set to F12/v. 

[0169]A shutter of a pixel by the side of the leftmost of frame #n-1 in drawing 40 
opens, an ingredient of the 2nd foreground of shutter time/v is set to F1 1/v, a shutter 
of the 2nd pixel opens it from the left in drawing 40 , and an ingredient of the 3rd 
foreground of shutter time/v is also set to F11/v. A shutter of the 3rd pixel opens 
from the left in drawing 40 , and an ingredient of the 4th foreground of shutter time/v 
is set to F1 1/v. 

[0170]A shutter of a pixel by the side of the leftmost of frame #n~1 in drawing 40 
opens, an ingredient of the 3rd foreground of shutter time/v is set to F10/v, a shutter 
of the 2nd pixel opens it from the left in drawing 40 , and an ingredient of the 4th 
foreground of shutter time/v is also set to F10/v. A shutter of a pixel by the side of 
the leftmost of frame #n-1 in drawing 40 opens, and an ingredient of the 4th 
foreground of shutter time/v is set to F09/v. 

[0171]Since an object corresponding to a background is standing it still, a shutter of 
the 2nd pixel opens from the left of frame #n-1 in drawing 40 , and an ingredient of the 



background of the first shutter time/v is set to B01/v. A shutter of the 3rd pixel 
opens from the left of frame #n-1 in drawing 40 , and an ingredient of the background 
of the beginning and the 2nd shutter time / v is set to B02/v. A shutter of the 4th 
pixel opens from the left of frame #n-1 in drawing 40 , and an ingredient of the 
background of the beginning thru/or the 3rd shutter time / v is set to B03/v. 
[0172]In frame #n-1 in drawing 40 , a pixel of most left-hand side belongs to a 
foreground region, and the 2nd thru/or the 4th pixel belong to a mixing zone which is a 
covered background region from left-hand side. 

[01 73]The 5th pixel thru/or the 1 2th pixel belong to a background region from the left 
of frame #n-1 in drawing 40 , and the pixel value is set to B04 thru/or B11, 
respectively. 

[01 74]The 1 st pixel thru/or the 5th pixel belong to a foreground region from the left of 
frame #n in drawing 40 . Ingredients of a foreground of the shutter time/v in a 
foreground region of frame #n are either F05/v thru/or F12/v. 

[0175]Since it moves so that an object corresponding to a foreground may be a rigid 
body, and can assume that it moves at uniform velocity and a picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, A shutter of the 
5th pixel opens from the left of frame #n in drawing 40 , an ingredient of a foreground of 
the first shutter time/v is set to F12/v, a shutter of the 6th pixel opens it from the left 
■ n drawing 40 , and an ingredient of the 2nd foreground of shutter time/v is also set to 
F12/v. A shutter of the 7th pixel opens from the left in drawing 40 , a shutter of the 8th 
pixel opens from an ingredient of the 3rd foreground of shutter time/v, and the left in 
drawing 40 , and an ingredient of the 4th foreground of shutter time/v is set to F12/v. 
[0176]A shutter of the 5th pixel opens from the left of frame #n in drawing 40 , an 
ingredient of the 2nd foreground of shutter time/v is set to F1 1/v, a shutter of the 6th 
pixel opens it from the left in drawing 40 , and an ingredient of the 3rd foreground of 
shutter time/v is also set to F11/v. A shutter of the 7th pixel opens from the left in 
drawing 40 , and an ingredient of the 4th foreground of shutter time/v is set to F1 1/v. 
[0177]A shutter of the 5th pixel opens from the left of frame #n in drawing 40 , an 
ingredient of the 3rd foreground of shutter time/v is set to F10/v, a shutter of the 6th 
pixel opens it from the left in drawing 40 , and an ingredient of the 4th foreground of 
shutter time/v is also set to F10/v. A shutter of the 5th pixel opens from the left of 
frame #n in drawing 40 , and an ingredient of the 4th foreground of shutter time/v is 
set to F09/v. 

[0178]Since an object corresponding to a background is standing it still, a shutter of 
the 6th pixel opens from the left of frame #n in drawing 40 , and an ingredient of the 
background of the first shutter time/v is set to B05/v. A shutter of the 7th pixel 
opens from the left of frame #n in drawing 40 , and an ingredient of the background of 
the beginning and the 2nd shutter time / v is set to B06/v. A shutter of the 8th pixel 
opens from the left of frame #n in drawing 40 , and an ingredient of the background of 



the beginning thru/or the 3rd shutter time / v is set to B07/v. 

[0179]In frame #n in drawing 40 , the 6th thru/or the 8th pixel belong to a mixing zone 
which is a covered background region from left-hand side. 

[0180]The 9th pixel thru/or the 12th pixel belong to a background region from the left 
of frame #n in drawing 40 , and a pixel value is set to B08 thru/or B1 1, respectively. 
[0181]The 1st pixel thru/or the 9th pixel belong to a foreground region from the left of 
frame #n+1 in drawing 40 . Ingredients of a foreground of the shutter time/v in a 
foreground region of frame #n+1 are either F01/v thru/or F12/v. 

[0182]Since it moves so that an object corresponding to a foreground may be a rigid 
body, and can assume that it moves at uniform velocity and a picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, A shutter of the 
9th pixel opens from the left of frame #n+1 in drawing 40 , an ingredient of a foreground 
of the first shutter time/v is set to F12/v, a shutter of the 10th pixel opens it from the 
left in d rawing 40 , and an ingredient of the 2nd foreground of shutter time/v is also set 
to F12/v. A shutter of the 1 1th pixel opens from the left in drawing 40 , a shutter of the 
12th pixel opens from an ingredient of the 3rd foreground of shutter time/v, and the 
left in d rawing 40 , and an ingredient of the 4th foreground of shutter time/v is set to 
F12/v. 

[0183]A shutter of the 9th pixel opens from the left of frame #n+1 in drawing 40 , an 
ingredient of a foreground of the 2nd period of shutter time/v is set to F11/v, a 
shutter of the 10th pixel opens it from the left in drawing 40 , and an ingredient of the 
3rd foreground of shutter time/v is also set to F11/v. A shutter of the 11th pixel 
opens from the left in drawing 40 , and an ingredient of the 4th foreground of shutter 
time/v is set to F11/v. 

[0184]A shutter of the 9th pixel opens from the left of frame #n+1 in drawing 40 , an 
ingredient of the 3rd foreground of shutter time/v is set to F10/v, a shutter of the 
10th pixel opens it from the left in drawing 40 , and an ingredient of the 4th foreground 
of shutter time/v is also set to F1 0/v. A shutter of the 9th pixel opens from the left of 
frame #n+1 in drawing 40 , and an ingredient of the 4th foreground of shutter time/v is 
set to F09/v. 

[0185]Since an object corresponding to a background is standing it still, a shutter of 
the 10th pixel opens from the left of frame #n+1 in drawing 40 , and an ingredient of the 
background of the first shutter time/v is set to B09/v. A shutter of the 11th pixel 
opens f^om the left of frame #n+1 in drawing 40 , and an ingredient of the background 
of the beginning and the 2nd shutter time / v is set to B10/v. A shutter of the 12th 
pixel opens from the left of frame #n+1 in drawing 40 , and an ingredient of the 
background of the beginning thru/or the 3rd shutter time / v is set to B1 1/v. 
[0186]In frame #n+1 in drawing 40 , the 10th thru/or the 12th pixel correspond to a 
mixing zone which is a covered background region from left-hand side. 
[0187] Drawing 41 is a model figure of a picture which extracted an ingredient of a 



foreground from a pixel value shown in drawing 40 . 

[0188]D rawing 42 is a pixel which three frames of a picture which picturized a 
foreground corresponding to an object which moves to right-hand side in a figure 
adjoined, and has been located in a line with one row with a stationary background, and 
is the model figure which developed a pixel value of a pixel of the same position to a 
time direction on a frame. An uncovered background region is included in drawing 42 . 
[0189]In drawing 42 , it can be assumed that an object corresponding to a foreground 
is a rigid body, and is moving at uniform velocity. Since an object corresponding to a 
foreground is moving so that it may be displayed on right-hand side by 4 pixels in the 
following frame, movement quantity v is 4. 

[0190]For example, a shutter of a pixel by the side of the leftmost of frame #n-1 in 
drawing 42 opens, the first ingredient of a foreground of shutter time/v is set to F13/v, 
a shutter of the 2nd pixel opens it from the left in drawing 42 , and an ingredient of the 
2nd foreground of shutter time/v is also set to F1 3/v. A shutter of the 3rd pixel opens 
from the left in drawing 42 , a shutter of the 4th pixel opens from an ingredient of the 
3rd foreground of shutter time/v, and the left in drawing 42 , and an ingredient of the 
4th foreground of shutter time/v is set to F13/v. 

[0191]A shutter of the 2nd pixel opens from the left of frame #n-1 in drawing 42 , an 
ingredient of a foreground of the first shutter time/v is set to F14/v, a shutter of the 
3rd pixel opens it from the left in drawing 42 , and an ingredient of the 2nd foreground 
of shutter time/v is also set to F14/v. A shutter of the 3rd pixel opens from the left in 
drawing 42 , and the first ingredient of a foreground of shutter time/v is set to F15/v. 
[01 92]Since an object corresponding to a background is standing it still, a shutter of a 
pixel by the side of the leftmost of frame #n-1 in drawing 42 opens, and an ingredient 
of the 2nd thru/or the 4th background of shutter time/v is set to B25/v. A shutter of 
the 2nd pixel opens from the left of frame #n-1 in drawing 42 , and an ingredient of the 
3rd and the 4th background of shutter time/v is set to B26/v. A shutter of the 3rd 
pixel opens from the left of frame #n-1 in drawing 42 , and an ingredient of the 4th 
background of shutter time/v is set to B27/v. 

[0193]In frame #n~1 in drawing 42 , a pixel of most left-hand side thru/or the 3rd pixel 
belong to a mixing zone which is an uncovered background region. 
[0194]The 4th pixel thru/or the 12th pixel belong to a foreground region from the left 
of frame #n-1 in drawing 42 . Ingredients of a foreground of a frame are either F13/v 
thru/or F24/v. 

[01 95]The 4th pixel belongs to a background region from a pixel thru/ or the left by the 
side of the leftmost of frame #n in drawing 42 , and a pixel value is set to B25 thru/or 
B28, respectively. 

[0196]Since it moves so that an object corresponding to a foreground may be a rigid 
body, and can assume that it moves at uniform velocity and a picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, A shutter of the 



5th pixel opens from the left of frame #n in drawing 42 , an ingredient of a foreground of 
the first shutter time/v is set to F1 3/v, a shutter of the 6th pixel opens it from the left 
in drawing 42 , and an ingredient of the 2nd foreground of shutter time/v is also set to 
F13/v. A shutter of the 7th pixel opens from the left in drawing 42 , a shutter of the 8th 
pixel opens from an ingredient of the 3rd foreground of shutter time/v, and the left in 
drawing 42 , and an ingredient of the 4th foreground of shutter time/v is set to F13/v. 
[0197]A shutter of the 6th pixel opens from the left of frame #n in drawing 42 , an 
ingredient of a foreground of the first shutter time/v is set to F14/v, a shutter of the 
7th pixel opens it from the left in drawing 42 , and an ingredient of the 2nd foreground 
of shutter time/v is also set to F1 4/v. A shutter of the 8th pixel opens from the left in 
drawing 42 , and an ingredient of a foreground of the first shutter time/v is set to 
F15/v. 

[0198]Since an object corresponding to a background is standing it still, a shutter of 
the 5th pixel opens from the left of frame #n in drawing 42 , and an ingredient of the 
2nd thru/or the 4th background of shutter time/v is set to B29/v. A shutter of the 6th 
pixel opens from the left of frame #n in drawing 42 , and an ingredient of the 3rd and 
the 4th background of shutter time/v is set to B30/v. A shutter of the 7th pixel opens 
from the left of frame #n in drawing 42 , and an ingredient of the 4th background of 
shutter time/v is set to B31/v. 

[0199]In frame #n in drawing 42 , the 5th pixel thru/or the 7th pixel belong to a mixing 
zone which is an uncovered background region from the left. 

[0200]The 8th pixel thru/or the 12th pixel belong to a foreground region from the left 
of frame #n in drawing 42 . Values corresponding to a period of shutter time/v in a 
foreground region of frame #n are either F13/v thru/or F20/v. 

[0201 ]The 8th pixel belongs to a background region from a pixel thru/or the left by the 
side of the leftmost of frame #n+1 in drawing 42 , and a pixel value is set to B25 
thru/or B32, respectively. 

[0202]Since it moves so that an object corresponding to a foreground may be a rigid 
body, and can assume that it moves at uniform velocity and a picture of a foreground 
may be displayed on 4-pixel right-hand side in the following frame, A shutter of the 
9th pixel opens from the left of frame #n+1 in drawing 42 , an ingredient of a foreground 
of the first shutter time/v is set to F13/v, a shutter of the 10th pixel opens it from the 
left in d rawing 42 , and an ingredient of the 2nd foreground of shutter time/v is also set 
to F13/v. A shutter of the 1 1th pixel opens from the left in drawing 42 , a shutter of the 
12th pixel opens from an ingredient of the 3rd foreground of shutter time/v, and the 
left in d rawing 42 , and an ingredient of the 4th foreground of shutter time/v is set to 
F13/v. 

[0203]A shutter of the 10th pixel opens from the left of frame #n+1 in drawing 42 , an 
ingredient of a foreground of the first shutter time/v is set to F14/v, a shutter of the 
1 1th pixel opens it from the left in drawing 42 , and an ingredient of the 2nd foreground 



of shutter time/v is also set to F14/v. A shutter of the 12th pixel opens from the left 
in drawing 42 , and an ingredient of a foreground of the first shutter time/v is set to 
F15/v. 

[0204]Since an object corresponding to a background is standing it still, a shutter of 
the 9th pixel opens from the left of frame #n+1 in drawing 42 , and an ingredient of the 
2nd thru/or the 4th background of shutter time/v is set to B33/v. A shutter of the 
10th pixel opens from the left of frame #n+1 in drawing 42 , and an ingredient of the 3rd 
and the 4th background of shutter time/v is set to B34/v. A shutter of the 1 1th pixel 
opens from the left of frame #n+1 in drawing 42 , and an ingredient of the 4th 
background of shutter time/v is set to B35/v. 

[0205]In frame #n+1 in drawing 42 , the 9th pixel thru/or the 11th pixel belong to a 
mixing zone which is an uncovered background region from the left. 
[0206]The 12th pixel belongs to a foreground region from the left of frame #n+1 in 
drawing 42 . Ingredients of a foreground of the shutter time/v in a foreground region of 
frame #n+1 are either F13/v thru/or F16/v. 

[0207]D rawing 43 is a model figure of a picture which extracted an ingredient of a 
foreground from a pixel value shown in drawing 42 . 

[0208]Return to drawing 27 and a pixel value of two or more frames is used for the 
field specific part 103, A flag which shows that it belongs to a foreground region, a 
background region, a covered background region, or an uncovered background region 
is matched for every pixel, and the mixture ratio calculation part 104 and the motion 
dotage controller 106 are supplied as area information. 

[0209]the mixture ratio which the mixture ratio calculation part 104 computed the 
mixture ratio alpha for every pixel about a pixel contained in a mixing zone based on a 
pixel value of two or more frames, and area information, and was computed — alpha is 
supplied to the foreground background separation part 105. 

[0210]the foreground background separation part 105 — a pixel value of two or more 
frames, area information, and the mixture ratio — extracting a foreground ingredient 
picture which consists only of an ingredient of a foreground based on alpha — a 
motion — a Japanese quince — the controller 106 is supplied. 

[021 1]A foreground ingredient picture to which the motion dotage controller 106 was 
supplied from the foreground background separation part 105, A foreground ingredient 
picture which was included in a foreground ingredient picture, moved based on a 
motion vector supplied from the motion detection part 102 and area information 
supplied from the field specific part 103, adjusted quantity of a Japanese quince, 
moved, and adjusted quantity of a Japanese quince is outputted. 

[0212]With reference to a flow chart of drawing 44 , processing of adjustment of 
quantity by the separation server 1 1 which moves and fades is explained. In Step S1 1, 
the field specific part 103 performs processing of field specification which generates 
area information which shows whether it belongs for every pixel of an inputted image 



to either a foreground region, a background region, a covered background region or an 
uncovered background region based on an inputted image. Details of processing of 
field specification are mentioned later. The field specific part 103 supplies generated 
area information to the mixture ratio calculation part 104. 

[021 3]In Step S11, the field specific part 103, It may be made to generate area 
information which shows whether it belongs for every pixel of an inputted image to 
either a foreground region, a background region or a mixing zone (distinction of a 
covered background region or an uncovered background region is not carried out) 
based on an inputted image. In this case, the foreground background separation part 
105 and the motion dotage controller 106 judge whether a mixing zone is a covered 
background region or it is an uncovered background region based on the direction of a 
motion vector. For example, when having ranked with a foreground region, a mixing 
zone and a background region, and order corresponding to the direction of a motion 
vector, the mixing zone, When it is judged with a covered background region and has 
ranked with a background region, a mixing zone and a foreground region, and order 
corresponding to the direction of a motion vector, the mixing zone is judged to be an 
uncovered background region. 

[0214]In Step S12, the mixture ratio calculation part 104 computes the mixture ratio 
alpha for every pixel contained in a mixing zone based on an inputted image and area 
information. Details of processing of mixture ratio calculation are mentioned later, the 
mixture ratio which the mixture ratio calculation part 104 computed — alpha is 
supplied to the foreground background separation part 105. 

[0215]in Step S13 — the foreground background separation part 105 — area 
information and the mixture ratio — based on alpha, an ingredient of a foreground is 
extracted from an inputted image, it moves as a foreground ingredient picture, and the 
Japanese quince controller 106 is supplied. 

[021 6]In Step S14, move and the Japanese quince controller 106, It is the continuous 
pixel located in a line in the motion direction based on a motion vector and area 
information, A batch which shows a position on a picture of a thing belonging to either 
an uncovered background region, a foreground region and a covered background 
region is generated, and it is contained in a foreground ingredient corresponding to a 
batch, and moves, and quantity of a Japanese quince is adjusted. Details of processing 
of adjustment of quantity of motion dotage are mentioned later. 

[0217]In Step S15, the separation server 11 judges whether processing was ended 
about the whole screen, and when judged with not having ended processing about the 
whole screen, it progresses to Step S14 and it repeats processing of adjustment of 
quantity which moves and fades for an ingredient of a foreground corresponding to a 
batch. 

[021 8]In Step S15, when judged with having ended processing about the whole screen, 
processing is ended. 



[0219]Thus, the separation server 11 separates a foreground and a background, can 
be contained in a foreground, can be moved, and can adjust quantity of a Japanese 
quince. That is, the separation server 1 1 can be contained in sample data which is a 
pixel value of a pixel of a foreground, can be moved, and can adjust quantity of a 
Japanese quince. 

[0220]Hereafter, each composition of the field specific part 103, the mixture ratio 
calculation part 104, the foreground background separation part 105, and the motion 
dotage controller 106 is explained. 

[0221] Drawing 45 is a block diagram showing an example of composition of the field 
specific part 103. The field specific part 103 which shows drawing 45 composition 
does not use a motion vector. The frame memory 201 memorizes an inputted picture 
per frame. Frame #n~1, frame #n whose frame memory 201 is a frame in front of [ of 
frame #n-2 which is a frame in front of / of frame #n / two, and frame #n ] one when 
an object of processing is frame #n, Frame #n+2 which is a frame after two, frame 
#n+1 which is a frame after [ of frame #n ] one, and frame #n, is memorized. 
[0222]A pixel value of a pixel of frame #n+2 in the same position as a position on a 
picture of a pixel which is an object of field specification [ the static/dynamic 
detection portion 202-1 ] of frame #n, And an absolute value of a difference of a read 
pixel value is computed by reading a pixel value of a pixel of frame #n+1 in the same 
position as a position on a picture of a pixel which is an object of field specification of 
frame #n from the frame memory 201. It judges whether the static/dynamic detection 
portion 202-1 has an absolute value of a difference of a pixel value of frame #n+2, and 
a pixel value of frame #n+1 larger than threshold Th set up beforehand, and when 
judged with an absolute value of a difference being larger than threshold Th, 
static/dynamic detection which shows a motion is supplied to the area judgment part 
203-1 . When judged with an absolute value of a difference of a pixel value of a pixel of 
frame #n+2 and a pixel value of a pixel of frame #n+1 being below threshold Th, the 
static/dynamic detection portion 202-1 supplies static/dynamic detection which 
shows stillness to the area judgment part 203-1. 

[0223]The static/dynamic detection portion 202-2 computes an absolute value of a 
difference of a pixel value by reading a pixel value of a pixel of frame #n+1 in the same 
position as a position on a picture of a pixel which is an object of field specification of 
frame #n, and a pixel value of a pixel which is the target of frame #n from the frame 
memory 201. The static/dynamic detection portion 202-2 an absolute value of a 
difference of a pixel value of frame #n+1 and a pixel value of frame #n, When it judges 
whether it is larger than threshold Th set up beforehand and judges that an absolute 
value of a difference of a pixel value is larger than threshold Th, static/dynamic 
detection which shows a motion is supplied to the area judgment part 203-1 and the 
area judgment part 203-2. When it judges that an absolute value of a difference of a 
pixel value of a pixel of frame #n+1 and a pixel value of a pixel of frame #n is below 



threshold Th, the static/dynamic detection portion 202-2 supplies static/dynamic 
detection which shows stillness to the area judgment part 203-1 and the area 
judgment part 203-2. 

[0224]The static/dynamic detection portion 202-3 computes an absolute value of a 
difference of a pixel value by reading a pixel value of a pixel of frame #n-1 in the same 
position as a position on a pixel value of a pixel which is an object of field specification 
of frame #n, and a picture of a pixel which is an object of field specification of frame #n 
from the frame memory 201. The static/dynamic detection portion 202-3 an absolute 
value of a difference of a pixel value of frame #n and a pixel value of frame #n-1, When 
it judges whether it is larger than threshold Th set up beforehand and judges that an 
absolute value of a difference of a pixel value is larger than threshold Th, 
static/dynamic detection which shows a motion is supplied to the area judgment part 
203-2 and the area judgment part 203-3. When it judges that an absolute value of a 
difference of a pixel value of a pixel of frame #n and a pixel value of a pixel of frame 
#n-1 is below threshold Th, the static/dynamic detection portion 202-3 supplies 
static/dynamic detection which shows stillness to the area judgment part 203-2 and 
the area judgment part 203-3. 

[0225]A pixel value of a pixel of frame #n-1 in the same position as a position on a 
picture of a pixel which is an object of field specification [ the static/dynamic 
detection portion 202-4 ] of frame #n, And an absolute value of a difference of a pixel 
value is computed by reading a pixel value of a pixel of frame #n-2 in the same 
position as a position on a picture of a pixel which is an object of field specification of 
frame #n from the frame memory 201 . The static/dynamic detection portion 202-4 an 
absolute value of a difference of a pixel value of frame #n-1 and a pixel value of frame 
#n-2, When it judges whether it is larger than threshold Th set up beforehand and 
judges that an absolute value of a difference of a pixel value is larger than threshold 
Th, static/dynamic detection which shows a motion is supplied to the area judgment 
part 203-3. When it judges that an absolute value of a difference of a pixel value of a 
pixel of frame #n-1 and a pixel value of a pixel of frame #n~2 is below threshold Th, 
the static/dynamic detection portion 202-4 supplies static/dynamic detection which 
shows stillness to the area judgment part 203-3. 

[0226]When static/dynamic detection with which static/dynamic detection supplied 
from the static/dynamic detection portion 202-1 showed stillness, and the area 
judgment part 203-1 was supplied from the static/dynamic detection portion 202-2 
shows a motion, "1" which shows that it belongs to an uncovered background region 
is set as an uncovered background region decision flag corresponding to a pixel judge 
[ pixel ] with a pixel which is an object of field specification in frame #n belonging to an 
uncover ed background region, and a field is judged to be. 

[0227]The area judgment part 203-1. [ whether static/dynamic detection supplied 
from the static/dynamic detection portion 202-1 shows a motion, and ] Or when 



static/dynamic detection supplied from the static/dynamic detection portion 202-2 
shows stillness, "0" which shows not belonging to an uncovered background region is 
set as an uncovered background region decision flag corresponding to a pixel judge 
[ pixel ] with a pixel which is an object of field specification in frame #n not belonging 
to an uncovered background region, and a field is judged to be. 

[0228]The area judgment part 203-1 supplies an uncovered background region 
decision flag with which "1 " or "0" was set up in this way to the decision flag storing 
frame memory 204. 

[0229]When static/dynamic detection with which static/dynamic detection supplied 
from the static/dynamic detection portion 202-2 showed stillness, and the area 
judgment part 203-2 was supplied from the static/dynamic detection portion 202-3 
shows stillness, "1" which shows that it belongs to a static region is set as a static 
region decision flag corresponding to a pixel judge [ pixel ] with a pixel which is an 
object of field specification in frame #n belonging to a static region, and a field is 
judged to be. 

[0230]The area judgment part 203-2. [ whether static/dynamic detection supplied 
from the static/dynamic detection portion 202-2 shows a motion, and ] Or when 
static/dynamic detection supplied from the static/dynamic detection portion 202-3 
shows a motion, "0" which shows not belonging to a static region is set as a static 
region decision flag corresponding to a pixel judge [ pixel ] with a pixel which is an 
object of field specification in frame #n not belonging to a static region, and a field is 
judged to be. 

[0231 ]The area judgment part 203-2 supplies a static region decision flag with which 
"1" or "0" was set up in this way to the decision flag storing frame memory 204. 
[0232]When static/dynamic detection with which static/dynamic detection supplied 
from the static/dynamic detection portion 202-2 showed a motion, and the area 
judgment part 203-2 was supplied from the static/dynamic detection portion 202-3 
shows a motion, "1" which shows that it belongs to a motion field is set as a motion 
area judgment flag corresponding to a pixel a pixel which is an object of field 
specification in frame #n moves, and judge [ pixel ] with belonging to a field and with 
which a field is judged to be. 

[0233]The area judgment part 203-2. [ whether static/dynamic detection supplied 
from the static/dynamic detection portion 202-2 shows stillness, and ] Or when 
static/dynamic detection supplied from the static/dynamic detection portion 202-3 
shows stillness, "0" which shows not belonging to a motion field is set as a motion 
area judgment flag corresponding to a pixel a pixel which is an object of field 
specification in frame #n moves, and judge [ pixel ] with not belonging to a field and 
with which a field is judged to be. 

[0234]The area judgment part 203-2 supplies a motion area judgment flag with which 
"1" or "0" was set up in this way to the decision flag storing frame memory 204. 



[0235]When static/dynamic detection with which static/dynamic detection supplied 
from the static/dynamic detection portion 202-3 showed a motion, and the area 
judgment part 203-3 was supplied from the static/dynamic detection portion 202-4 
shows stillness, "1" which shows that it belongs to a covered background region is set 
as a covered background region decision flag corresponding to a pixel judge [ pixel ] 
with a pixel which is an object of field specification in frame #n belonging to a covered 
background region, and a field is judged to be. 

[0236]The area judgment part 203-3. [ whether static/dynamic detection supplied 
from the static/dynamic detection portion 202-3 shows stillness, and ] Or when 
static/dynamic detection supplied from the static/dynamic detection portion 202-4 
shows a motion, "0" which shows not belonging to a covered background region is set 
as a covered background region decision flag corresponding to a pixel judge [ pixel ] 
with a pixel which is an object of field specification in frame #n not belonging to a 
covered background region, and a field is judged to be. 

[0237]The area judgment part 203-3 supplies a covered background region decision 
flag with which "1" or "0" was set up in this way to the decision flag storing frame 
memory 204. 

[0238]An uncovered background region decision flag with which the decision flag 
storing frame memory 204 was supplied from the area judgment part 203-1, A static 
region decision flag supplied from the area judgment part 203-2, a motion area 
judgment flag supplied from the area judgment part 203-2, and a covered background 
region decision flag supplied from the area judgment part 203-3 are memorized, 
respectively. 

[0239]The decision flag storing frame memory 204 supplies a memorized uncovered 
background region decision flag, a static region decision flag, a motion area judgment 
flag, and a covered background region decision flag to the synchronizer 205. . The 
synchronizer 205 was supplied from the decision flag storing frame memory 204. An 
uncovered background region decision flag, a static region decision flag, Based on a 
motion area judgment flag and a covered background region decision flag, Each pixel 
generates area information which shows that it belongs to either an uncovered 
background region, a static region, a motion field and a covered background region, 
and supplies the decision flag storing frame memory 206. 

[0240]The decision flag storing frame memory 206 memorizes area information 
supplied from the synchronizer 205, and it outputs memorized area information. 
[0241]Next, an example of processing of the field specific part 103 is explained with 
reference to drawing 46 thru/or drawing 50 . 

[0242]While an object corresponding to a foreground is moving, a position on a screen 
of a picture corresponding to an object changes for every frame. As shown in drawing 
46 , in frame #n, a picture corresponding to an object located in a position shown by Yn 
(x, y) is located in Yn+1 (x, y) in frame #n+1 which is the following frame. 



[0243]A model figure which developed a pixel value of a pixel which adjoins in the 
motion direction of a picture corresponding to an object of a foreground, and is 
located in a line with one row to a time direction is shown in drawing 22 . For example, 
a model figure [ in / when level / to a screen / in the motion direction of a picture 
corresponding to an object of a foreground / drawing 47 ] shows a model which 
developed a pixel value of an adjoining pixel on one line to a time direction. 
[0244]In drawing 47 , a line in frame #n is the same as a line in frame #n+1. 
[0245]In frame #n, an ingredient of a foreground corresponding to an object contained 
in the 2nd pixel thru/or the 1 3th pixel from the left is contained in the 6th thru/or the 
17th pixel from the left in frame #n+1. 

[0246]In frame #n, pixels belonging to a covered background region are the 11th 
thru/or the 13th pixel from the left, and pixels belonging to an uncovered background 
region are the 2nd thru/or the 4th pixel from the left. In frame #n+1, pixels belonging 
to a covered background region are the 15th thru/or the 17th pixel from the left, and 
pixels belonging to an uncovered background region are the 6th thru/or the 8th pixel 
from the left. 

[0247]In an example shown in drawing 47 , since 4 pixels of ingredients of a foreground 
included in frame #n are moving in frame #n+1 , movement quantity v is 4. The virtual 
number of partitions corresponds to movement quantity v, and is 4. 
[0248]Next, change of a pixel value of a pixel belonging to a mixing zone before and 
behind a frame currently observed is explained. 

[0249]In frame #n which is shown in drawing 48 and whose movement quantity v of a 
foreground a background stands it still and is 4, pixels belonging to a covered 
background region are the 15th thru/or the 17th pixel from the left. Since movement 
quantity v is 4, in frame #n-1 in front of one, the 1 5th thru/or the 1 7th pixel belong to 
a background region only including an ingredient of a background from the left. In 
frame #n-2 in front of one, the 1 5th thru/or the 1 7th pixel belong to a background 
region only including an ingredient of a background from the left. 

[0250]Since an object corresponding to a background is standing it still here, a pixel 
value of the left of frame #n-1 to the 15th pixel does not change from a pixel value of 
the left of frame #n-2 to the 15th pixel. Similarly, a pixel value of the left of frame 
#n-1 to the 16th pixel does not change from a pixel value of the left of frame #n-2 to 
the 16th pixel, and a pixel value of the left of frame #n~1 to the 17th pixel does not 
change from a pixel value of the left of frame #n-2 to the 17th pixel. 
[0251]That is, since a pixel of frame #n-1 corresponding to a pixel belonging to a 
covered background region in frame #n and frame #n-2 comprises only an ingredient 
of a background and a pixel value does not change, an absolute value of the difference 
turns into about 0 value. Therefore, static/dynamic detection to a pixel of frame #n-1 
corresponding to a pixel belonging to a mixing zone in frame #n and frame #n-2 is 
judged by the static/dynamic detection portion 202-4 to be stillness. 



[0252]Since a pixel belonging to a covered background region in frame #n contains an 
ingredient of a foreground, a pixel value differs from a case where only an ingredient of 
a background in frame #n-1 is comprised. Therefore, static/dynamic detection to a 
pixel belonging to a mixing zone in frame #n and a pixel of corresponding frame #n-1 is 
judged by the static/dynamic detection portion 202-3 to be a motion. 
[0253]Thus, the area judgment part 203-3 judges with a corresponding pixel belonging 
to a covered background region, when a result of static/dynamic detection which 
shows static/dynamic detection portion 202-3 lost motion is supplied and a result of 
static/dynamic detection which shows stillness from the static/dynamic detection 
portion 202-4 is supplied. 

[0254]In frame #n which is shown in drawing 49 and whose movement quantity v of a 
foreground a background stands it still and is 4, pixels contained in an uncovered 
background region are the 2nd thru/or the 4th pixel from the left. Since movement 
quantity v is 4, in frame #n+1 after one, the 2nd thru/or the 4th pixel belong to a 
background region only including an ingredient of a background from the left. In frame 
#n+2 after one, the 2nd thru/or the 4th pixel belong to a background region only 
including an ingredient of a background from the left. 

[0255]Since an object corresponding to a background is standing it still here, a pixel 
value of the left of frame #n+2 to the 2nd pixel does not change from a pixel value of 
the left of frame #n+1 to the 2nd pixel. Similarly, a pixel value of the left of frame #n+2 
to the 3rd pixel does not change from a pixel value of the left of frame #n+1 to the 3rd 
pixel, and a pixel value of the left of frame #n+2 to the 4th pixel does not change from 
a pixel value of the left of frame #n+1 to the 4th pixel. 

[0256]That is, since a pixel of frame #n+1 corresponding to a pixel belonging to an 
uncovered background region in frame #n and frame #n+2 comprises only an 
ingredient of a background and a pixel value does not change, an absolute value of the 
difference turns into about 0 value. Therefore, static/dynamic detection to a pixel of 
frame corresponding to a pixel belonging to a mixing zone in frame #n and frame 
#n+2 is judged by the static/dynamic detection portion 202-1 to be stillness. 
[0257]Since a pixel belonging to an uncovered background region in frame #n contains 
an ingredient of a foreground, a pixel value differs from a case where only an 
ingredient of a background in frame #n+1 is comprised. Therefore, static/dynamic 
detection to a pixel belonging to a mixing zone in frame #n and a pixel of corresponding 
frame #n+1 is judged by the static/dynamic detection portion 202-2 to be a motion. 
[0258]Thus, the area judgment part 203-1 judges with a corresponding pixel belonging 
to an uncovered background region, when a result of static/dynamic detection which 
shows static/dynamic detection portion 202-2 lost motion is supplied and a result of 
static/dynamic detection which shows stillness from the static/dynamic detection 
portion 202-1 is supplied. 

[0259]D rawing 50 is a figure showing criteria of the field specific part 103 in frame #n. 



A pixel of frame #n-2 in the same position as a position on a picture of a pixel which is 
the target of a judgment of frame #n, A pixel of frame #n-1 in the same position as a 
position on a picture of a pixel which a pixel of frame #n-1 in the same position as a 
position on a picture of a pixel which is the target of a judgment of frame #n is judged 
to be stillness, and is the target of a judgment of frame #n, When a pixel of frame #n is 
judged to be a motion, the field specific part 103 judges with a pixel which is the target 
of a judgment of frame #n belonging to a covered background region. 
[0260]A pixel of frame #n-1 in the same position as a position on a picture of a pixel 
which is the target of a judgment of frame #n, When a pixel of frame #n is judged to be 
stillness and a pixel of frame #n+1 in the same position as a position on a picture of a 
pixel of frame #n and a pixel which is the targets of a judgment of frame #n is judged to 
be stillness, The field specific part 103 judges with a pixel which is the target of a 
judgment of frame #n belonging to a static region. 

[0261 ]A pixel of frame #n-1 in the same position as a position on a picture of a pixel 
which is the target of a judgment of frame #n, When a pixel of frame #n is judged to be 
a motion and a pixel of frame #n+1 in the same position as a position on a picture of a 
pixel of frame #n and a pixel which is the targets of ajudgment of frame #n is judged to 
be a motion, It judges with a pixel which is the target of a judgment of frame #n moving, 
and the field specific part 103 belonging to a field. 

[0262]A pixel of frame #n+1 in the same position as a position on a picture of a pixel of 
frame #n and a pixel which is the targets of ajudgment of frame #n is judged to be a 
motion, A pixel of frame #n+1 in the same position as a position on a picture of a pixel 
which is the target of ajudgment of frame #n, When a pixel of frame #n+2 in the same 
position as a position on a picture of a pixel which is the target of ajudgment of frame 
#n is judged to be stillness, the field specific part 103 judges with a pixel which is the 
target of ajudgment of frame #n belonging to an uncovered background region. 
[0263]D rawing 51 is a figure showing an example of a specific result of a field of the 
field specific part 103. In drawing 51 (A), a pixel judged that belongs to a covered 
background region is displayed in white. In drawing 51 (B), a pixel judged that belongs 
to an uncovered background region is displayed in white. 

[0264]In drawing 51 (C), a pixel judged that belongs to a motion field is displayed in 
white. In drawing 51 (D), a pixel judged that belongs to a static region is displayed in 
white. 

[0265]D rawing 52 is a figure showing as a picture area information which shows a 
mixing zone among area information which the decision flag storing frame memory 206 
outputs. In drawing 52 , a pixel judged that belongs to a covered background region or 
an uncovered background region, i.e., a pixel judged that belong to a mixing zone, is 
displayed in white. Area information which shows a mixing zone which the decision flag 
storing frame memory 206 outputs shows a portion with a texture surrounded by 
portion without a texture in a mixing zone and a foreground region. 



[0266]Next, with reference to a flow chart of drawing 53 , processing of field 
specification of the field specific part 103 is explained. In Step S201, the frame 
memory 201 acquires a picture of frame #n-2 containing frame #n which is the target 
of a judgment thru/or frame #n+2. 

[0267]In Step S202, the static/dynamic detection portion 202-3, When it judges 
whether it is stillness and is judged with stillness by pixel of frame #n-1, and a pixel of 
the same position of frame #n, it progresses to Step S203, and the static/dynamic 
detection portion 202-2 is with a pixel of frame #n, and a pixel of the same position of 
frame #n+1, and judges whether it is stillness. 

[0268]By pixel of frame #n, and a pixel of the same position of frame #n+1, when 
judged with stillness, progress to Step S204 in Step S203, and the area judgment part 
203-2, '1" which shows that it belongs to a static region is set as a static region 
decision flag corresponding to a pixel a field is judged to be. The area judgment part 
203-2 supplies a static region decision flag to the decision flag storing frame memory 
204, and procedure follows it to Step S205. 

[0269]When it is judged with a motion by pixel of frame #n-1, and a pixel of the same 
position of frame #n in Step S202, Or in Step S203, since a pixel of frame #n does not 
belong to a static region when judged with a motion by pixel of frame #n, and a pixel of 
the same position of frame #n+1 , processing of Step S204 is skipped and procedure 
progresses to Step S205. 

[0270]In Step S205, the static/dynamic detection portion 202-3, When it judges 
whether it is a motion and is judged with a motion by pixel of frame #n-1, and a pixel of 
the same position of frame #n, it progresses to Step S206, and the static/dynamic 
detection portion 202-2 is with a pixel of frame #n, and a pixel of the same position of 
frame #n+1, and judges whether it is a motion. 

[0271]By pixel of frame #n, and a pixel of the same position of frame #n+1, when 
judged with a motion, progress to Step S207 in Step S206, and the area judgment part 
203-2, "1" which shows that it belongs to a motion field is set as a motion area 
judgment flag corresponding to a pixel a field is judged to be. The area judgment part 
203-2 supplies a motion area judgment flag to the decision flag storing frame memory 
204, and procedure follows it to Step S208. 

[0272]When it is judged with stillness by pixel of frame #n-1, and a pixel of the same 
position of frame #n in Step S205, Or in Step S206, since a pixel of frame #n moves by 
pixel of frame #n, and a pixel of the same position of frame #n+1 and it does not belong 
to a field by them when judged with stillness, processing of Step S207 is skipped and 
procedure progresses to Step S208. 

[0273]In Step S208, the static/dynamic detection portion 202-4, When it judges 
whether it is stillness and is judged with stillness by pixel of frame #n-2, and a pixel of 
the sarre position of frame #n~1, it progresses to Step S209, and the static/dynamic 
detection portion 202-3 is with a pixel of frame #n-1, and a pixel of the same position 



of frame #n, and judges whether it is a motion. 

[0274]By pixel of frame #n~1, and a pixel of the same position of frame #n ? when 
judged v/ith a motion, progress to Step S210 in Step S209, and the area judgment part 
203-3, "1" which shows that it belongs to a covered background region is set as a 
covered background region decision flag corresponding to a pixel a field is judged to be. 
The area judgment part 203-3 supplies a covered background region decision flag to 
the decision flag storing frame memory 204, and procedure follows it to Step S21 1. 
[0275]When it is judged with a motion by pixel of frame #n-2, and a pixel of the same 
position of frame #n-1 in Step S208, Or in Step S209 by pixel of frame #n-1, and a 
pixel of the same position of frame #n. Since a pixel of frame #n does not belong to a 
covered background region when judged with stillness, processing of Step S210 is 
skipped and procedure progresses to Step S21 1. 

[0276]In Step S211, the static/dynamic detection portion 202-2, When it judges 
whether it is a motion and is judged with a motion by pixel of frame #n, and a pixel of 
the same position of frame #n+1, it progresses to Step S212, and the static/dynamic 
detection portion 202-1 is with a pixel of frame #n+1, and a pixel of the same position 
of frame #n+2, and judges whether it is stillness. 

[0277]By pixel of frame #n+1, and a pixel of the same position of frame #n+2, when 
judged with stillness, progress to Step S213 in Step S212, and the area judgment part 
203-1, "1" which shows that it belongs to an uncovered background region is set as 
an uncovered background region decision flag corresponding to a pixel a field is judged 
to be. The area judgment part 203-1 supplies an uncovered background region 
decision flag to the decision flag storing frame memory 204, and procedure follows it 
to Step S214. 

[0278]When it is judged with stillness by pixel of frame #n, and a pixel of the same 
position of frame #n+1 in Step S211, Or in Step S212 by pixel of frame #n+1, and a 
pixel of the same position of frame #n+2. Since a pixel of frame #n does not belong to 
an uncovered background region when judged with a motion, processing of Step S213 
is skipped and procedure progresses to Step S214. 

[0279]In Step S214, the field specific part 103, When it judges whether a field was 
pinpointed about all the pixels of frame #n and is judged with pinpointing a field about 
no pixels of frame #n, procedure returns to Step S202 and repeats processing of field 
specification about other pixels. 

[0280]When judged with having pinpointed a field about all the pixels of frame #n in 
Step S214, progress to Step S215 and the synchronizer 205, An uncovered 
background region decision flag memorized by the decision flag storing frame memory 
204, And area information which shows a mixing zone is generated based on a covered 
background region decision flag, Each pixel generates area information which shows 
that it belongs to either an uncovered background region, a static region, a motion 
field and a covered background region, generated area information is set as the 



decision flag storing frame memory 206, and processing is ended. 

[0281]Thus, the field specific part 103 can generate area information which shows 

that it belongs to a motion field, a static region, an uncovered background region, or a 

covered background region about each of a pixel contained in a frame. 

[0282]When the field specific part 103 applies logical sum to area information 

corresponding to an uncovered background region and a covered background region, 

Area information corresponding to a mixing zone is generated, and it may be made to 

generate area information which comprises a flag which shows that it belongs to a 

motion field, a static region, or a mixing zone about each of a pixel contained in a 

frame. 

[0283]When an object corresponding to a foreground has a texture, the field specific 
part 103 can be moved more correctly and can pinpoint a field. 

[0284]The field specific part 103 can output area information which shows a static 
region as area information which shows a foreground region for area information which 
shows a motion field as area information which shows a background region. 
[0285]Although it explained that an object corresponding to a background was 
standing it still, processing which pinpoints a field mentioned above even if a picture 
corresponding to a background region included a motion is applicable. For example, 
while a picture corresponding to a background region is moving uniformly, the field 
specific part 103 shifts the whole picture corresponding to this motion, and is 
processed like a case where an object corresponding to a background is standing it 
still. When a picture corresponding to a background region includes a different motion 
for every part, the field specific part 103 chooses a pixel corresponding to a motion, 
and performs above-mentioned processing. 

[0286]D rawing 54 is a block diagram showing other examples of composition of the 
field specific part 103. The field specific part 103 shown in drawing 54 does not use a 
motion vector. The background image generation part 301 generates a background 
image corresponding to an inputted image, and supplies a generated background image 
to the binary object image extraction part 302. The background image generation part 
301 extracts an image object corresponding to an object of a background included in 
an inputted image, for example, and generates a background image. 
[0287]An example of a model figure which developed a pixel value of a pixel which 
adjoins in the motion direction of a picture corresponding to an object of a foreground, 
and is located in a line with one row to a time direction is shown in drawing 55 . For 
example, a model figure [ in / when level / to a screen / in the motion direction of a 
picture corresponding to an object of a foreground / drawing 55 ] shows a model 
which developed a pixel value of an adjoining pixel on one line to a time direction. 
[0288]In drawing 55 , a line in frame #n is the same as a line in frame #n-1 and frame 
#n+1. 

[0289]In frame #n, in frame #n-1 f an ingredient of a foreground corresponding to an 



object contained in the 6th pixel thru/or the 17th pixel from the left is contained in 
the 2nd thru/or the 13th pixel from the left, and is contained in the 10th thru/or the 
21st pixel from the left in frame #n+1. 

[0290]In frame #n-1, pixels belonging to a covered background region are the 11th 
thru/or the 13th pixel from the left, and pixels belonging to an uncovered background 
region are the 2nd thru/or the 4th pixel from the left. In frame #n, pixels belonging to a 
covered background region are the 15th thru/or the 17th pixel from the left, and pixels 
belonging to an uncovered background region are the 6th thru/or the 8th pixel from 
the left. In frame #n+1, pixels belonging to a covered background region are the 19th 
thru/or the 21st pixel from the left, and pixels belonging to an uncovered background 
region are the 10th thru/or the 12th pixel from the left. 

[0291]In frame #n-1, pixels belonging to a background region are the 14th thru/or the 
21st pixel from the 1st pixel and the left from the left. In frame #n, pixels belonging to 
a background region are the 18th thru/or the 21st pixel from the 1st thru/or the 5th 
pixel, and the left from the left. In frame #n+1, pixels belonging to a background region 
are the 1st thru/or the 9th pixel from the left. 

[0292]An example of a background image corresponding to an example of drawing 55 
which the background image generation part 301 generates is shown in drawing 56 . A 
background image comprises a pixel corresponding to an object of a background, and 
does not contain an ingredient of a picture corresponding to an object of a foreground. 
[0293]Based on correlation of a background image and an inputted image, the binary 
object image extraction part 302 generates a binary object image, and supplies a 
generated binary object image to the temporal change primary detecting element 303. 
[0294] Drawing 57 is a block diagram showing composition of the binary object image 
extraction part 302. The correlation value operation part 321 calculates correlation of 
a background image and an inputted image which were supplied from the background 
image generation part 301, and supplies a correlation value which generated and 
generated a correlation value to the threshold treating part 322. 

[0295]As are shown in drawing 58 (A), and it is indicated in drawing 58 (B) as a block in 
a background image of 3x3 centering on X 4 , the correlation value operation part 321, 
for example, A correlation value corresponding to Y 4 is computed by applying a 
formula (4) to a block in an inputted image of 3x3 centering on Y 4 corresponding to a 
block in a background image. 



[0296] 
[Equation 2] 



8 
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[Equation 3] 

X = ~ (5) 



[Equation 4] 
s 

Y= (6) 



[0297]The correlation value operation part 321 supplies a correlation value computed 
in this way corresponding to each pixel to the threshold treating part 322. 
[0298]As shown in drawing 59 (A), the correlation value operation part 321, for 
example, It may be made to compute a difference absolute value corresponding to Y 4 
by applying a formula (7) to a block in a background image of 3x3 centering on X 4 , and 
a block in an inputted image of 3x3 centering on Y 4 corresponding to [ as shown in 
drawing 59 (B) ] a block in a background image. 
[0299] 
[Equation 5] 

mmmnmm=t\{Xi-Yi)\ (?) 

i=0 



[0300]The correlation value operation part 321 is supplied to the threshold treating 
part 322 by making into a correlation value a difference absolute value computed in 
this way. 

[0301 ]The threshold treating part 322 compares a pixel value of a correlation picture 
with threshold thO, When a correlation value is the zero or less threshold th, 1 is set as 
a pixel value of a binary object image, when a correlation value is larger than threshold 
thO, 0 is set as a pixel value of a binary object image, and 0 or 1 outputs a binary 
object image set as a pixel value. It may be made for the threshold treating part 322 to 
use threshold thO which it may be made to memorize threshold thO beforehand, or was 
inputted from the outside. 

[0302]D rawing 60 is a figure showing an example of a binary object image 
corresponding to a model of an inputted image shown in drawing 55 . In a binary object 
image, 0 is set to a high pixel of a background image and correlation at a pixel value. 
[0303]D rawing 61 is a block diagram showing composition of the temporal change 
primary detecting element 303. The frame memory 341 memorizes a binary object 
image of frame #n-1 supplied from binary object image extraction part 302, frame #n, 
and frame #n+1, when judging a field about a pixel of frame #n. 



[0304]Based on a binary object image of frame #n-1 memorized by frame memory 341, 
frame #n, and frame #n+1, the area judgment part 342 judges a field about each pixel 
of frame #n, and outputs area information which generated and generated area 
information. 

[0305]D rawing 62 is a figure explaining a judgment of the area judgment part 342. 
When a pixel which is observing a binary object image of frame #n is 0, the area 
judgment part 342 judges with a pixel which is observing frame #n belonging to a 
background region. 

[0306]When a pixel which is observing a binary object image of frame #n is 1, a pixel to 
which a binary object image of frame #n-1 corresponds is 1 and a pixel to which a 
binary object image of frame #n+1 corresponds is 1, The area judgment part 342 
judges with a pixel which frame #n is observing belonging to a foreground region. 
[0307]A pixel which is observing a binary object image of frame #n is 1, and when a 
pixel to which a binary object image of frame #n-1 corresponds is 0, the area judgment 
part 342 judges with a pixel which is observing frame #n belonging to a covered 
background region. 

[0308]A pixel which is observing a binary object image of frame #n is 1, and when a 
pixel to which a binary object image of frame #n+1 corresponds is 0, the area judgment 
part 342 judges with a pixel which is observing frame #n belonging to an uncovered 
background region. 

[0309]D rawing 63 is a figure showing an example which the temporal change primary 
detecting element 303 judged about a binary object image corresponding to a model of 
an inputted image shown in drawing 55 . Since a pixel to which frame #n of a binary 
object image corresponds is 0, the temporal change primary detecting element 303 
judges that the 1 st thru/or the 5th pixel belong to a background region from the left of 
frame #n. 

[031 0]A pixel of frame #n of a binary object image is 1, and since a pixel to which 
frame #n+1 corresponds is 0, the temporal change primary detecting element 303 
judges that the 6th thru/or the 9th pixel belong to an uncovered background region 
from the left. 

[031 1]A pixel of frame #n of a binary object image is 1, a pixel to which frame #n-1 
corresponds is 1 , and since a pixel to which frame #n+1 corresponds is 1 , the temporal 
change primary detecting element 303 judges that the 10th thru/or the 13th pixel 
belong to a foreground region from the left. 

[031 2]A pixel of frame #n of a binary object image is 1, and since a pixel to which 
frame #n-1 corresponds is 0, the temporal change primary detecting element 303 
judges that the 14th thru/or the 17th pixel belong to a covered background region 
from the left. 

[0313]Since a pixel to which frame #n of a binary object image corresponds is 0, the 
temporal change primary detecting element 303 judges that the 18th thru/or the 21st 



pixel belong to a background region from the left. 

[0314]Next, with reference to a flow chart of drawing 64 , processing of field 
specification of the area judgment part 103 is explained. In Step S301, the background 
image generation part 301 of the area judgment part 103 extracts an image object 
corresponding to an object of a background included in an inputted image based on an 
inputted image, for example, and supplies a background image which generated and 
generated a background image to the binary object image extraction part 302. 
[0315]In Step S302, the binary object image extraction part 302 calculates a 
correlation value of an inputted image and a background image supplied from the 
background image generation part 301 by the operation explained with reference to 
drawing 58 , for example. In Step S303, the binary object image extraction part 302 
calculates a binary object image from correlation value and threshold thO, for example 
by comparing a correlation value with threshold thO. 

[0316]In Step S304, the temporal change primary detecting element 303 performs 
processing of an area judgment, and ends processing. 

[0317]With reference to a flow chart of drawing 65 , details of processing of an area 
judgment corresponding to Step S304 are explained. In Step S321, the area judgment 
part 342 of the temporal change primary detecting element 303, In [ in frame #n 
memorized by the frame memory 341, judge whether a pixel to observe is 0, and ] 
frame #n, When judged with a pixel to observe being 0, it progresses to Step S322, and 
if a pixel which frame #n observes belongs to a background region, it will set up, and 
processing is ended. 

[031 8]When judged with a pixel to observe being 1 in frame #n in Step S321, progress 
to Step S323 and the area judgment part 342 of the temporal change primary 
detecting element 303, In [ in frame #n memorized by the frame memory 341 , a pixel to 
observe is 1 , and ] frame #n-1 , In [judge whether a corresponding pixel is 0 and a pixel 
to observe is 1 in frame #n, and ] frame #n-1, When judged with a corresponding pixel 
being 0, it progresses to Step S324, and if a pixel which frame #n observes belongs to 
a covered background region, it will set up, and processing is ended. 
[031 9]Iri Step S323, in frame #n. [ whether a pixel to observe is 0 and ] Or when 
judged with a corresponding pixel being 1 in frame #n-1, progress to Step S325 and 
the area judgment part 342 of the temporal change primary detecting element 303, In 
[ in frame #n memorized by the frame memory 341 , a pixel to observe is 1 , and ] frame 
#n+1, In [judge whether a corresponding pixel is 0 and a pixel to observe is 1 in frame 
#n, and ] frame #n+1, When judged with a corresponding pixel being 0, it progresses to 
Step S326, and if a pixel which frame #n observes belongs to an uncovered 
background region, it will set up, and processing is ended. 

[0320]In Step S325, in frame #n. [ whether a pixel to observe is 0 and ] Or in frame 
#n+1, when judged with a corresponding pixel being 1, it progresses to Step S327, and 
the area judgment part 342 of the temporal change primary detecting element 303 



sets a pixel which frame #n observes to a foreground region, and ends processing. 
[0321]Thus, based on a correlation value of a picture as which the field specific part 
103 was inputted, and a corresponding background image, It can be specified whether 
a pixel of an inputted image belongs to either a foreground region, a background region, 
a covered background region and an uncovered background region, and area 
information corresponding to a specified result can be generated. 
[0322]D rawing 66 is a block diagram showing other composition of the field specific 
part 103. The field specific part 103 shown in drawing 66 uses a motion vector 
supplied from the motion detection part 102, and its position information. The same 
number is given to the same portion as a case where it is shown in drawing 54 , and the 
explanation is omitted. 

[0323]The robust-ized part 361 generates a binary object image made robust based 
on a binary object image of N frames supplied from the binary object image extraction 
part 302, and outputs it to it in the temporal change primary detecting element 303. 
[0324]D rawing 67 is a block diagram explaining composition of the robust-ized part 
361. The motion compensation section 381 outputs a binary object image which 
compensated a motion of a binary object image of N frames and with which a motion 
was compensated to the switch 382 based on a motion vector supplied from the 
motion detection part 102, and its position information. 

[0325]With reference to an example of drawing 68 and drawing 69 , a motion 
compensation of the motion compensation section 381 is explained. For example, 
frame #n-1, frame #n which show drawing 68 an example when judging a field of frame 
#n, And when a binary object image of frame #n+1 is inputted, the motion 
compensation section 381, Based on a motion vector supplied from the motion 
detection part 102, as an example is shown in drawing 69 , the motion compensation of 
a binary object image of frame #n-1 and the binary object image of frame #n+1 is 
carried out, and a binary object image by which the motion compensation was carried 
out is supplied to the switch 382. 

[0326]The switch 382 outputs a binary object image in which the motion 
compensation of the 1st frame was carried out to the frame memory 383-1, and 
outputs a binary object image in which the motion compensation of the 2nd frame was 
carried out to the frame memory 383-2. Similarly the switch 382 outputs each of a 
binary object image in which the motion compensation of the 3rd thru/or the N-1st 
frames was carried out to either the frame memory 383-3 thru/or frame memory 383- 
(N-1), A binary object image in which the motion compensation of the Nth frame was 
carried out is outputted to frame memory 383-N. 

[0327]The frame memory 383-1 memorizes a binary object image in which the motion 
compensation of the 1st frame was carried out, and outputs a binary object image 
memorized to the weighting section 384-1. The frame memory 383-2 memorizes a 
binary object image in which the motion compensation of the 2nd frame was carried 



out, and outputs a binary object image memorized to the weighting section 384-2. 
[0328]Similarly each of the frame memory 383-3 thru/or frame memory 383- (N-1), 
Either of the binary object images in which the motion compensation of the 3rd frame 
thru/or the N-1st frames was carried out is memorized, and a binary object image 
memorized is outputted to either the weighting section 384-3 thru/or weighting 
section 384- (N-1). Frame memory 383-N memorizes a binary object image in which 
the motion compensation of the Nth frame was carried out, and outputs a binary 
object image memorized to weighting section 384-N. 

[0329]The weighting section 384-1 multiplies by the dignity w1 beforehand provided in 
a pixel value of a binary object image in which the motion compensation of the 1st 
frame supplied from the frame memory 383-1 was carried out, and supplies it to the 
integrating part 385. The weighting section 384-2 multiplies by the dignity w2 
beforehand provided in a pixel value of a binary object image in which the motion 
compensation of the 2nd frame supplied from the frame memory 383-2 was carried 
out, and supplies it to the integrating part 385. 

[0330]Similarly each of the weighting section 384-3 thru/or weighting section 384- 
(N-1), It multiplies by either of the dignity w3 thru/or dignity w (N-1) beforehand 
provided in a pixel value of a binary object image in which the motion compensation of 
the 3rd supplied from either the frame memory 383-3 thru/or frame memory 383- 
(N-1) thru/or the N-1st ones of the frames was carried out, The integrating part 385 
is supplied. Weighting section 384-N multiplies by the dignity wN beforehand provided 
in a pixel value of a binary object image in which the motion compensation of the Nth 
frame supplied from frame memory 383-N was carried out, and supplies it to the 
integrating part 385. 

[0331]The integrating part 385 generates a binary object image by comparing with 
threshold thO by which the motion compensation of the 1 thru/or Nth frame was 
carried out, and either of the dignity w1 thru/or wN was able to multiply, respectively 
and which integrated a pixel value to which a binary object image corresponds, and 
defined an integrated pixel value beforehand. 

[0332]Thus, since the robust-ized part 361 generates a binary OBUJIETO picture 
made robust from N binary object images and supplies it to the temporal change 
primary detecting element 303, The field specific part 103 which shows drawing 66 
composition can pinpoint a field more correctly as compared with a case where it is 
shown in drawing 54 , even if a noise is contained in an inputted image. 
[0333]Next, processing of field specification of the field specific part 103 which shows 
composition in drawing 66 is explained with reference to a flow chart of drawing 70 . 
Since processing of Step S341 thru/or Step S343 is the same as that of Step S301 
thru/or Step S303 explained with a flow chart of drawing 64 respectively, the 
explanation is omitted. 

[0334]In Step S344, the robust-ized part 361 performs processing of robust-izing. 



[0335]In Step S345, the temporal change primary detecting element 303 performs 
processing of an area judgment, and ends processing. Since details of processing of 
Step S345 are the same as that of processing explained with reference to a flow chart 
of drawing 65 , the explanation is omitted. 

[0336] Next, with reference to a flow chart of drawing 71 , details of processing of 
robust-izing corresponding to processing of Step S344 of drawing 70 are explained. In 
Step S361, the motion compensation section 381 performs processing of a motion 
compensation of a binary object image inputted based on a motion vector supplied 
from the motion detection part 102, and its position information. In Step S362, either 
the frame memory 383-1 thru/or 383-N memorize a binary object image which was 
supplied via the switch 382 and by which the motion compensation was carried out. 
[0337]Ir Step S363, the robust-ized part 361, When it judges whether N binary object 
images were memorized and is judged with N binary object images not being 
memorized, it returns to Step S361 and processing of a motion compensation of a 
binary object image and processing of memory of a binary object image are repeated. 
[0338]In Step S363, when judged with N binary object images having been memorized, 
it progresses to Step S364, and each of the weighting section 384-1 thru/or 384-N 
multiplies by it and carries out weighting of the dignity of either w1 thru/or wN to each 
of N binary object images. 

[0339]In Step S365, the integrating part 385 integrates N binary object images by 
which weighting was carried out. 

[0340]In Step S366, by comparison with threshold th1 defined beforehand, etc., the 
integrat ng part 385 generates a binary object image from an integrated picture, and 
ends processing, for example. 

[0341]Thus, the field specific part 103 which shows drawing 66 composition can 
generate area information based on a binary object image made robust. 
[0342] As mentioned above, the field specific part 103 can generate area information 
which shows that it belongs to a motion field, a static region, an uncovered 
background region, or a covered background region about each of a pixel contained in 
a frame. 

[0343]D rawing 72 is a block diagram showing an example of composition of the 
mixture ratio calculation part 104. Based on an inputted image, by the operation 
corresponding to a model of a covered background region, the presumed mixture ratio 
treating part 401 computes the presumed mixture ratio for every pixel, and supplies 
the computed presumed mixture ratio to the mixture ratio deciding part 403. 
[0344]Based on an inputted image, by the operation corresponding to a model of an 
uncovered background region, the presumed mixture ratio treating part 402 computes 
the presumed mixture ratio for every pixel, and supplies the computed presumed 
mixture ratio to the mixture ratio deciding part 403. 

[0345]Since it can assume that an object corresponding to a foreground is moving at 



uniform velocity in shutter time, as for the mixture ratio alpha of a pixel belonging to a 
mixing zone, it has the following character. That is, the mixture ratio alpha changes 
linearly corresponding to change of a position of a pixel. If change of a position of a 
pixel is made into one dimension, change of the mixture ratio alpha can be expressed 
in a straight line, and if change of a position which is a pixel is made into two 
dimensions, change of the mixture ratio alpha can be expressed at a flat surface. 
[0346]S nce a period of one frame is short, an object corresponding to a foreground is 
a rigid body, and assumption will be realized if it is moving at uniform velocity. 
[0347]In this case, inclination of the mixture ratio alpha serves as a reciprocal ratio of 
movement quantity v within shutter time of a foreground. 

[0348]An example which is the ideal mixture ratio alpha is shown in drawing 73 . The 
inclination I in a mixing zone which is the ideal mixture ratio alpha can be expressed as 
a reciprocal of movement quantity v. 

[0349]it is shown in drawing 73 — as — the ideal mixture ratio — in a background 
region, alpha has a value of 1, in a foreground region, it has a value of 0, exceeds 0 in 
a mixing zone, and has less than one value. 

[0350]Ir! an example of drawing 74 , the pixel value C06 of the 7th pixel can be 
expressed with a formula (8) using the pixel value P06 of the left of frame #n-1 to the 
7th pixel from the left of frame #n. 
[0351] 
[Equation 6] 



[0352]In a formula (8), the pixel value M and the pixel value P06 of a pixel of a mixing 
zone are expressed for the pixel value C06 as the pixel value B of the pixel of a 
background region. That is, the pixel value M of the pixel of a mixing zone and the pixel 
value B of the pixel of a background region can be expressed like a formula (9) and a 
formula (10), respectively. 
[0353] 
M=C06 (9) 
B=P06 (10) 

[0354]2/v in a formula (8) correspond to the mixture ratio alpha. Since movement 
quantity v is 4, the mixture ratio alpha of the 7th pixel is set to 0.5 from the left of 
frame #n. 

[0355]As mentioned above, the formula (3) showing the mixture ratio alpha by 
considering that the pixel value C of frame #n currently observed is a pixel value of a 



C06 == B06/v + B06/v + FOliv + F02fv 
== P06lv + P06iv + FOl/v + FOliv 



2 




1=1 



mixing zone, and considering that the pixel value P of frame #n~1 in front of frame #n 
is a pixel value of a background region is rewritten like a formula (1 1). 



f of a formula (11) is peace sigma;Fi/v of an ingredient of a foreground included in a 
pixel currently observed. A variable included in a formula (1 1) is two, the mixture ratio 
alpha and the sum f of an ingredient of a foreground. 

[0357]Similarly, movement quantity v in an uncovered background region is 4, and the 
virtual number of partitions of a time direction shows drawing 75 a model which is 4 
and which developed a pixel value to a time direction. 

[0358]In an uncovered background region, like expression in a covered background 
region mentioned above, A formula (3) showing the mixture ratio alpha by considering 
that the pixel value C of frame #n currently observed is a pixel value of a mixing zone, 
and considering that the pixel value N of frame #n+1 after frame #n is a pixel value of 
a background region can be expressed like a formula (12). 
[0359] 

C=alpha-N+f(12) 

[0360]Although it explained that an object of a background was standing it still, when 
an object of a background is moving, a formula (8) thru/or a formula (12) can be 
applied by using a pixel value of a pixel of a position made to correspond to movement 
quantity v of a background. For example, in drawing 74 , movement quantity v of an 
object corresponding to a background is 2, and when the virtual number of partitions is 
2, while an object corresponding to a background is moving to right-hand side in a 
figure, let the pixel value B of a pixel of a background region in a formula (10) be the 
pixel value P04. 

[0361]Since a formula (11) and a formula (12) include two variables, respectively, they 
cannot ask for the mixture ratio alpha as it is. Here, a picture serves as the almost 
same pixel value by the pixels which approach since correlation is generally spatially 
strong. 

[0362]Then, since a foreground ingredient has spatially strong correlation, it 
transforms a formula so that the sum f of an ingredient of a foreground can be drawn 
from a frame of before or the back, and asks for the mixture ratio alpha. 
[0363]The pixel value Mc of the 7th pixel can be expressed with a formula (13) from 
the left of frame #n of drawing 76 . 



[0356] 

C=alpha-P+f (11) 



[0364] 
[Equation 7] 




(13) 



2/v of the 1st paragraph of the right-hand side of a formula (13) are equivalent to the 



mixture ratio alpha. Suppose that the 2nd paragraph of the right-hand side of a 
formula (13) is expressed like a formula (14) using the pixel value of next frame #n+1. 
[0365] 
[Equation 8] 

12 W 

Y,Filv = &-1£Filv (14) 

i=U i = 7 

[0366]Here, suppose that a formula (15) is materialized using the spatial correlations 

of the ingredient of a foreground. 

[0367] 

F=F05=F06=F07=F08=F09=F1 0=F1 1 =F1 2 (1 5) 

A formula (14) can be replaced like a formula (16) using a formula (15). 

[0368] 

[Equation 9] 
12 ~ 

2 Filv = 4r 

= £• -p- -F (16) 

beta can be expressed with a formula (17) as a result. 
[0369] 

beta = two fourths (1 7) 

[0370]If it generally assumes that an ingredient of a foreground related to a mixing 
zone is equal as shown in a formula (15), a formula (18) will be materialized from a 
relation of an internal ratio about all the pixels of a mixing zone. 
[0371] 

beta=1 -alpha (18) 

[0372]A formula (18) is materialized, then a formula (1 1) can be developed as shown in 

a formula (1 9). 

[0373] 

[Equation 10] 

C = a-P+f 

T+V-l 



[0374]Similarly, a formula (18) is materialized, then a formula (12) can be developed as 

shown in a formula (20). 

[0375] 



[Equation 1 1] 



C = a*N+f 



a)*2lFifv 



i = r 



= a-N+(l 



at)-p 



(20) 



[0376]In a formula (19) and a formula (20), since C, N, and P are known pixel values, a 
variable included in a formula (19) and a formula (20) is the mixture ratio alpha. A 
relation of C in a formula (19) and a formula (20), N, and P is shown in drawing 77 . C is 
a pixel value of a pixel which frame #n is observing which computes the mixture ratio 
alpha. N is a pixel value which are a pixel currently observed and a pixel of frame #n+1 
to which a position of the direction of space corresponds. P is a pixel value which are 
a pixel currently observed and a pixel of frame #n-1 to which a position of the 
direction of space corresponds. 

[0377]Therefore, since one variable will be included in each of a formula (19) and a 
formula (20), the mixture ratio alpha is computable using a pixel value of a pixel of 
three frames. Conditions for the right mixture ratio alpha to be computed by solving a 
formula (19) and a formula (20), In an image object of a foreground picturized while an 
ingredient of a foreground related to a mixing zone was equal, namely, an object of a 
foreground was standing it still, It is a pixel corresponding to the direction of a motion 
of an object of a foreground located in a boundary of an image object, and a pixel value 
of a pixel which one twice the number of movement quantity v is following is a fixed 
thing. 

[0378]As mentioned above, the mixture ratio alpha of a pixel belonging to a covered 
background region is computed by formula (21), and the mixture ratio alpha of a pixel 
belonging to an uncovered background region is computed by formula (22). 
[0379] 

alpha=(C-N)/(P-N) (21) 
alpha=(OP)/(N-P) (22) 

[0380]D rawing 78 is a block diagram showing composition of the presumed mixture 
ratio treating part 401. The frame memory 421 memorizes an inputted picture per 
frame, and supplies a frame after [ of a frame inputted as an inputted image ] one to 
the frame memory 422 and the mixture ratio operation part 423. 

[0381]The frame memory 422 memorizes an inputted picture per frame, and supplies a 
frame after [ of a frame currently supplied from the frame memory 421 ] one to the 
mixture ratio operation part 423. 

[0382]Therefore, when frame #n+1 is inputted into the mixture ratio operation part 
423 as an inputted image, the frame memory 421 supplies frame #n to the mixture 
ratio operation part 423, and the frame memory 422 supplies frame #n-1 to the 



mixture ratio operation part 423. 

[0383]The pixel value C of a pixel which frame #n is observing by the operation which 
shows the mixture ratio operation part 423 in a formula (21). The presumed mixture 
ratio of a pixel which is observing based on the pixel value P of a pixel currently 
observed, the pixel value N of a pixel of frame #n+1 to which a space position 
corresponds and a pixel currently observed, and a pixel of frame #n-1 to which a 
space position corresponds is computed, and the computed presumed mixture ratio is 
outputted. For example, while a background is standing it still, the mixture ratio 
operation part 423, the pixel value C of a pixel which frame #n is observing, and a 
position in a pixel currently observed and a frame — the same, the pixel value N of a 
pixel of frame #n+1, and a position in a pixel currently observed and a frame — the 
same — the presumed mixture ratio of a pixel currently observed is computed based 
on the pixel value P which is a pixel of frame #n-1, and the computed presumed 
mixture ratio is outputted to it. 

[0384]Thus, based on an inputted image, the presumed mixture ratio treating part 401 
can compute the presumed mixture ratio, and can supply it to the mixture ratio 
deciding part 403. 

[0385]By the operation the presumed mixture ratio treating part 401 indicates the 
presumed mixture ratio treating part 402 to be to a formula (21). Except for portions 
which compute the presumed mixture ratio of a pixel currently observed by the 
operation shown in a formula (22) differing to computing the presumed mixture ratio of 
a pixel currently observed, since it is the same as that of the presumed mixture ratio 
treating part 401, the explanation is omitted. 

[0386]D rawing 79 is a figure showing an example of the presumed mixture ratio 
computed by the presumed mixture ratio treating part 401. The presumed mixture 
ratio shown in drawing 79 shows a result in case movement quantity v of a foreground 
corresponding to an object which is moving at uniform velocity is 1 1 to one line. 
[0387]As the presumed mixture ratio is shown in drawing 73 in a mixing zone, it turns 
out that it is changing almost linearly. 

[0388]. Returned to drawing 72 and the mixture ratio deciding part 403 was supplied 
from the field specific part 103. A pixel which is the target of calculation which is the 
mixture ratio alpha sets up the mixture ratio alpha based on area information which 
shows whether it belongs to either a foreground region, a background region, a 
covered background region or an uncovered background region. The mixture ratio 
deciding part 403 sets 0 as the mixture ratio alpha, when the target pixel belongs to a 
foreground region, When the target pixel belongs to a background region, 1 is set as 
the mixture ratio alpha and the target pixel belongs to a covered background region, 
When the presumed mixture ratio supplied from the presumed mixture ratio treating 
part 401 is set as the mixture ratio alpha and the target pixel belongs to an uncovered 
background region, the presumed mixture ratio supplied from the presumed mixture 



ratio treating part 402 is set as the mixture ratio alpha, the mixture ratio which the 
mixture ratio deciding part 403 set up based on area information — alpha is outputted. 
[0389]D rawing 80 is a block diagram showing other composition of the mixture ratio 
calculation part 104. The selecting part 441 supplies a pixel of a next frame to the 
presumed mixture ratio treating part 442, before corresponding to a pixel belonging to 
a covered background region, and this based on area information supplied from the 
field specific part 103. The selecting part 441 supplies a pixel of a next frame to the 
presumed mixture ratio treating part 443, before corresponding to a pixel belonging to 
an uncovered background region, and this based on area information supplied from the 
field specific part 103. 

[0390]Based on a pixel value inputted from the selecting part 441, the presumed 
mixture ratio treating part 442 computes the presumed mixture ratio of a pixel 
currently observed belonging to a covered background region by the operation shown 
in a formula (21), and supplies the computed presumed mixture ratio to the selecting 
part 444. 

[0391]Based on a pixel value inputted from the selecting part 441, the presumed 
mixture ratio treating part 443 computes the presumed mixture ratio of a pixel 
currently observed belonging to an uncovered background region by the operation 
shown in a formula (22), and supplies the computed presumed mixture ratio to the 
selecting part 444. 

[0392]When the presumed mixture ratio which is 0 when the target pixel belongs to a 
foreground region based on area information supplied from the field specific part 103 
is chosen, it is set as the mixture ratio alpha and the target pixel belongs to a 
background region, the selecting part 444 chooses the presumed mixture ratio which 
is 1, and sets it as the mixture ratio alpha. When the target pixel [ selecting part / 
444 ] belongs to a covered background region, choosing the presumed mixture ratio 
supplied from the presumed mixture ratio treating part 442 — the mixture ratio — 
choosing the presumed mixture ratio supplied from the presumed mixture ratio 
treating part 443, when it is set as alpha and the target pixel belongs to an uncovered 
background region — the mixture ratio — it is set as alpha, the mixture ratio which 
chose the selecting part 444 based on area information, and was set up — alpha is 
outputted. 

[0393]thus, the mixture ratio computed by the mixture ratio calculation part 104 which 
has other composition shown in drawing 80 having computed the mixture ratio alpha 
for every pixel in which a picture is included — alpha can be outputted. 
[0394]With reference to a flow chart of drawing 81 , processing of calculation of the 
mixture ratio alpha of the mixture ratio calculation part 104 which shows composition 
in drawing 72 is explained. In Step S401, the mixture ratio calculation part 104 
acquires area information supplied from the field specific part 103. In Step S402, the 
presumed mixture ratio treating part 401 performs processing of an operation of the 



presumed mixture ratio with a model corresponding to a covered background region, 
and supplies the computed presumed mixture ratio to the mixture ratio deciding part 
403. Details of processing of an operation of mixture ratio presumption are later 
mentioned with reference to a flow chart of drawing 82 . 

[0395]In Step S403, the presumed mixture ratio treating part 402 performs processing 
of an operation of the presumed mixture ratio with a model corresponding to an 
uncovered background region, and supplies the computed presumed mixture ratio to 
the mixture ratio deciding part 403. 

[0396]In Step S404, the mixture ratio calculation part 104 judges whether the mixture 
ratio alpha was presumed about the whole frame, when judged with not presuming the 
mixture ratio alpha about the whole frame, returns to Step S402 and performs 
processing which presumes the mixture ratio alpha about the following pixel. 
[0397]About the whole frame, when judged with having presumed the mixture ratio 
alpha, progress to Step S405 in Step S404, and the mixture ratio deciding part 403, A 
pixel sets up the mixture ratio alpha based on area information supplied from the field 
specific part 103 which shows whether it belongs to either a foreground region, a 
background region, a covered background region or an uncovered background region. 
The mixture ratio deciding part 403 sets 0 as the mixture ratio alpha, when the target 
pixel belongs to a foreground region, When the target pixel belongs to a background 
region, I is set as the mixture ratio alpha and the target pixel belongs to a covered 
background region, When the presumed mixture ratio supplied from the presumed 
mixture ratio treating part 401 is set as the mixture ratio alpha and the target pixel 
belongs to an uncovered background region, the presumed mixture ratio supplied from 
the presumed mixture ratio treating part 402 is set as the mixture ratio alpha, and 
processing is ended. 

[0398]thus, the mixture ratio which is the characteristic quantity corresponding to 
each pixel area information to which the mixture ratio calculation part 104 was 
supplied from the field specific part 103, and based on an inputted image — alpha is 
computable. 

[0399]Since processing of calculation of the mixture ratio alpha of the mixture ratio 
calculation part 104 which shows composition in drawing 80 is the same as processing 
explained with a flow chart of drawing 81 , the explanation is omitted. 
[0400]IMext, processing of mixture ratio presumption by a model corresponding to a 
covered background region corresponding to Step S402 of drawing 81 is explained 
with reference to a flow chart of drawing 82 . 

[0401]In Step S421, the mixture ratio operation part 423 acquires the pixel value C of 
a noticed picture element of frame #n from the frame memory 421. 
[0402]In Step S422, the mixture ratio operation part 423 acquires the pixel value P of 
a pixel of frame #n-1 corresponding to a noticed picture element from the frame 
memory 422. 



[0403]In Step S423, the mixture ratio operation part 423 acquires the pixel value N of 
a pixel of frame #n+1 corresponding to a noticed picture element contained in an 
inputted image. 

[0404]In Step S424 r the mixture ratio operation part 423 calculates the presumed 
mixture ratio based on the pixel value C of a noticed picture element of frame #n, the 
pixel value P of a pixel of frame #n-1, and the pixel value N of a pixel of frame #n+1. 
[0405]In Step S425, the mixture ratio operation part 423, When it judges whether 
processing which calculates the presumed mixture ratio was ended about the whole 
frame and judged with not having ended processing which calculates the presumed 
mixture ratio about the whole frame, it returns to Step S421 and processing which 
computes the presumed mixture ratio about the following pixel is repeated. 
[0406]In Step S425, when judged with having ended processing which calculates the 
presumed mixture ratio about the whole frame, processing is ended. 
[0407]Thus, the presumed mixture ratio treating part 401 can calculate the presumed 
mixture ratio based on an inputted image. 

[0408]Since processing of mixture ratio presumption by a model corresponding to an 
uncovered background region in Step S403 of drawing 81 is the same as processing 
shown in a flow chart using a formula corresponding to a model of an uncovered 
background region of drawing 82 , the explanation is omitted. 

[0409]Since the same processing as a flow chart shown in drawing 82 is performed 
and the presumed mixture ratio is calculated, the presumed mixture ratio treating part 
442 and the presumed mixture ratio treating part 443 which are shown in drawing 80 
omit the explanation. 

[0410]the mixture ratio mentioned above even if a picture corresponding to a 
background region included a motion although it explained that an object 
corresponding to a background was standing it still — processing which asks for alpha 
is applicable. For example, while a picture corresponding to a background region is 
moving uniformly, the presumed mixture ratio treating part 401 shifts the whole 
picture corresponding to a motion of a background, and is processed like a case where 
an object corresponding to a background is standing it still. When a picture 
corresponding to a background region includes a motion of a different background for 
every part, the presumed mixture ratio treating part 401 chooses a pixel 
corresponding to a motion of a background as a pixel corresponding to a pixel 
belonging to a mixing zone, and performs above-mentioned processing. 
[041 1]The mixture ratio calculation part 104 performs only processing of mixture ratio 
presumption by a model corresponding to a covered background region about all the 
pixels, and it may be made to output the computed presumed mixture ratio as the 
mixture ratio alpha. In this case, the mixture ratio alpha shows a rate of an ingredient 
of a background about a pixel belonging to a covered background region, and shows a 
rate of an ingredient of a foreground about a pixel belonging to an uncovered 



background region, the mixture ratio computed in this way about a pixel belonging to 
an uncovered background region, if an absolute value of difference of alpha and 1 is 
computed and a computed absolute value is set as the mixture ratio alpha, the mixture 
ratio the separation server 1 1 indicates a rate of an ingredient of a background to be 
about a pixel belonging to an uncovered background region — it can ask for alpha. 
[0412]The mixture ratio calculation part 104 performs only processing of mixture ratio 
presumption by a model corresponding to an uncovered background region about all 
the pixels, and it may be made to output the computed presumed mixture ratio as the 
mixture ratio alpha similarly. 

[0413]next — using character in which the mixture ratio alpha changes linearly — the 
mixture ratio — the mixture ratio calculation part 104 which computes alpha is 
explained. 

[0414]As mentioned above, since a formula (11) and a formula (12) include two 
variables, respectively, they cannot ask for the mixture ratio alpha as it is. 
[0415]Then, [ in shutter time ] corresponding to change of a position of a pixel by an 
object corresponding to a foreground moving at uniform velocity, a formula which 
approximated the sum f of an ingredient of the mixture ratio alpha and a foreground in 
the direction of space is stood using character in which the mixture ratio alpha 
changes linearly. A formula which approximated the sum f of an ingredient of the 
mixture ratio alpha and a foreground is solved using plurality of a group of a pixel value 
of a pixel belonging to a pixel value and a background region of a pixel belonging to a 
mixing zone. 

[0416]When change which is the mixture ratio alpha is approximated as a straight line, 

the mixture ratio alpha is expressed with a formula (23). 

[0417] 

Alpha=il+p (23) 

In a formula (23), i is an index of the direction of space which set a position of a pixel 
currently observed to 0. I is a slope of a line of the mixture ratio alpha, p is the mixture 
ratio alpha of a pixel which is a section of a straight line of the mixture ratio alpha and 
which is both observed. In a formula (23), although the index i is known, the inclination 
I and its section p are strange. 

[0418]Relation between the index i, the inclination I, and the section p is shown in 
drawing 83 . 

[041 9]the mixture ratio — the mixture ratio from which plurality differs to two or more 
pixels by approximating alpha like a formula (23) — alpha is expressed by two 
variables. In an example shown in drawing 83 , the five mixture ratio to five pixels is 
expressed by the inclination I which is two variables, and the section p. 
[0420]That a picture is horizontal and when the mixture ratio alpha is approximated at 
a flat surface shown in drawing 84 , and the motion v corresponding to the two 
directions of vertical is taken into consideration, a formula (23) is extended to a flat 



surface and the mixture ratio alpha is expressed with a formula (24). 
[0421] 

Alpha=jm+kq+p (24) 

In a formula (24), j is the horizontal index which set a position of a pixel currently 

observed to 0, and k is a vertical index, m is horizontal inclination to a field of the 

mixture ratio alpha, and q is inclination to a perpendicular direction of a field of the 

mixture ratio alpha, p is a section of a field of the mixture ratio alpha. 

[0422]For example, in frame #n shown in drawing 74 , a formula (25) thru/or a formula 

(27) are materialized about COS thru/or C07, respectively. 

[0423] 

C05=alpha05andB05/v+f05 (25) 
C06=alpha06andB06/v+f06 (26) 
C07=alpha07andB07/v+f07 (27) 

[0424]An ingredient of a foreground is nearby in agreement, namely, if F01 thru/or F03 
are transposed to Fc, a formula (28) will be materialized noting that F01 thru/or F03 
are equal. 
[0425] 

f(x) = (1-alpha(x)) and Fc (28) 

In a formula (28), x expresses a position of the direction of space. 

[0426]If alpha(x) is replaced by a formula (24), a formula (28) can be expressed as a 

formula (29). 

[0427] 

= [ f(x) ] (1- (jm+kq+p)), Fc=j, +k, and (-q-Fc) +(1-p) (-Fc) =js+kt+u (29) 

[0428]In a formula (29), (-m-Fc), (-q-Fc), and (1-p) -Fc are replaced, as shown in a 

formula (30) thru/or a formula (32). 

[0429] 

s=-m-Fc (30) 

t=-q-Fc (31) 

u = (1-p) and Fc (32) 

[0430]In a formula (29), j is the horizontal index which set a position of a pixel 
currently observed to 0, and k is a vertical index. 

[0431]Thus, into shutter time, an object corresponding to a foreground moves at 
uniform velocity, and since assumption that an ingredient corresponding to a 
foreground is constant in the neighborhood is materialized, the sum of an ingredient of 
a foreground is approximated by a formula (29). 

[0432]When it approximates the mixture ratio alpha in a straight line, the sum of an 

ingredient of a foreground can be expressed with a formula (33). 

[0433] 

f(x)=is+u (33) 

[0434]When the mixture ratio alpha of a formula (13) and the sum of a foreground 



ingredient are replaced using a formula (24) and a formula (29), the pixel value M is 

expressed with a formula (34). 

[0435] 

M = (jm+kq+p) and B+js+kt+u=jB-m+kB-q+B-p+j-s+k-t+u (34) 

[0436]In a formula (34), a strange variable is six, the section p, s, and t of the 
horizontal inclination m to a field of the mixture ratio alpha, the inclination q to a 
perpendicular direction of a field of the mixture ratio alpha, and a field of the mixture 
ratio alpha, and u. 

[0437]The mixture ratio alpha is computed by making it correspond to a pixel near the 
pixel currently observed, and solving two or more normal equations which set the pixel 
value M or the pixel value B as a normal equation showing in an equation (34) and with 
which the pixel value M or the pixel value B was set as it with a least square method. 
[0438]For example, the horizontal index j of a pixel currently observed is set to 0, the 
vertical index k is set to 0, and if the pixel value M or the pixel value B is set as a 
normal equation showing in an equation (34) about a pixel of 3x3 near the pixel 
currently observed, an equation (35) thru/or an equation (43) will be obtained. 
[0439] 

M_ 1 and _r=(-1) -B H and --m+(-1) -B_ t H and q+B_ land _-p+(-1) -s+ (-1) and t+u (35) 

— 1=(0) and [ M Q and ] B0 and _ 1 _ m+( _ 1) _ B0 andq+B 0, and-1-p+(0), s+(-1), and t+u (36) 

M+1 and -1=(+1) -B+1 and -1-m+(-1) -B+1. -1 and q+B+1, and -1-p+(+1) -s+ (-1) and t+u (37) 

M-1. 0=(-1) -B-1, and 0-m+(0) and B-1. 0 and q+B-1, and 0-p+(-1) -s+(0) and t+u (38) 

MO, 0=(0) and BO, and 0-m+(0) and BO, Oandq+BO, and 0-p+(0), s+(0). and t+u (39) 

M+1. 0=(+1) -B+1, and 0-m+(0) and B+1, 0andq+B+1, and 0-p+(+1) -s+(0) and t+u (40) 

M-1 and +1=(-1) -B-1 and +1-m+(+1) -B-1. +1 and q+B-1, and +1-p+(-1) -s+ (+1) and t+u (41) 

+1 =: (0) and [ Mq and ] B0 +1 _ m+ ( +1 ) _ B0 +i an dq+B0, and +1-p+(0), s+ (+1), and t+u (42) 

M+1, +1=(+1) -B+1. +1-m+(+1) -B+1. +1andq+B+1. and +1-p+(+1) -s+ (+1) and t+u (43) 

[0440]Since the horizontal index j of a pixel currently observed is 0 and the vertical index k is 0, the mixture ratio alpha of a pixel currently observed 
is more nearly equal to the value p at the time of j= 0 and k= 0. i.e.. a section, than a formula (24). 

[0441] Therefore, what is necessary is to compute the horizontal inclination m, the inclination q to a perpendicular direction, the section p, s. and t, 
and each value of u, and just to output the section p as the mixture ratio alpha with a least square method, based on nine formulas of a formula (35) 
thru/or a formula (43). 

[0442]next — applying a least square method — the mixture ratio — a concrete procedure is explained rather than computing alpha. 
[0443]When the one index x expresses the index i and the index k, relation between the index i, the index k, and the index x is expressed with a 
formula (44). 
[0444] 

x=(j + D and3+ (k+1) (44) 

[0445]The horizontal inclination m, the inclination q to a perpendicular direction, the section p, s, and t, and u are expressed as the variable wO, w1, 
w2, w3, w4, and W5, respectively, and jB, kB, B, j, k, and 1 are expressed as aO, a1, a2. a3. a4. and a5. respectively. If the error ex is taken into 
consideration, a formula (35) thru/or a formula (43) can be rewritten at a ceremony (45). 
[0446] 



[Equation 12] 

Mx = y £ i dyWy + ex (45) 



In a formula 5), x is one value of the integers of 0 thru/or 8. 
[0447]A formila (46) can be drawn from a formula (45). 
[0448] 

[Equation 13] 

ex = Nlx-jtf f ayWy (46) 



[0449]Here, ir order to apply a least square method, the sum of squares E with error is defined as being shown in a formula (47). 
[0450] 

[Equation 14] 

E = 2 ex 2 (47) 

X-..0 



[0451]In ordei for an error to become the minimum, the partial differential of the variable Wv to the sum of squares E with error should just be set 

to 0. Here, v i i one value of the integers of 0 thru/or 5. Therefore, wy is calculated so that a formula (48) may be filled. 

[0452] 

[Equation 15] 



(48) 



[0453]A formula (49) will be obtained if a formula (46) is substituted for a formula (48). 
[0454] 

[Equation 16] 

S (tfv E ay • Wy) = 2 ay Mjc (49) 

[0455]wy is computed by sweeping out at six ceremony produced by substituting any one of the integers of 0 thru/or 5 for v of a formula (49), and 
applying law (elimination of Gauss-Jordan) etc. to it, for example. As mentioned above, wO is the horizontal inclination m, w1 is the inclination q to 
a perpendicul \r direction, w2 is the section p, w3 is s, w4 is t, and w5 is u. 

[0456]As mertioned above, it can ask for the horizontal inclination m, the inclination q to a perpendicular direction, the section p, s. and t, and u by 
applying a least square method to the formula which set up the pixel value M and the pixel value B. 

[0457]Althouj;h the pixel value of the pixel contained in a mixing zone was set to M in the explanation corresponding to a formula (35) thru/ or a 
formula (43) end the pixel value of the pixel contained in a background region was explained as B. When the pixel currently observed is contained in 



a covered background region, it is necessary to each in the case of being contained in an uncovered background region to stand a normal equation. 
[0458]For example, when asking for the mixture ratio alpha of a pixel contained in a covered background region of frame #n shown in drawing 74 , the 
pixel values PD4 thru/or P08 of a pixel of C04 thru/or C08, and frame #n-1 of a pixel of frame #n are set as a normal equation. 
[0459]When asking for the mixture ratio alpha of a pixel contained in an uncovered background region of frame #n shown in drawing 75 , the pixel 
values N28 th-u/or N32 of a pixel of C28 thru/or C32, and frame #n+1 of a pixel of frame #n are set as a normal equation. 

[0460]When, c omputing the mixture ratio alpha of a pixel contained in a covered background region shown in drawing 85 for example, the following 

formulas (50) thru/or formulas (58) is stood, the mixture ratio — a pixel value of a pixel which computes alpha is Mc5. 

[0461] 

Mc1 = (-1 ), B s1 , m+ (-1 ), Bc1 . q+Bd , p+ (-1 ), s+ (-1 ), and t+u (50) 
Mc2=(0), Bc2, m+ (-1), Bc2. q+Bc2, p+(0). s+ (-1), and t+u (51) 
Mc3 = (+1), B :3, m+ (-1), Bc3, q+Bc3, p+ (+1), s+ (-1), and t+u (52) 
Mc4=(-1), Bel, andm+ (0) -Bc4, q+Bc4, and p+(-1) -s+(0) and t+u (53) 
Mc5= (0) Bc5 q+Bc5, -Bc5andm+ (0), and p+(0), s+(0), and t+u (54) 
Mc6=(+1). Bcti, andm+ (0) -Bc6, q+Bc6, and p+(+1) -s+(0) and t+u (55) 
Mc7 = (-1), Bo7, m+ (+1), Bc7, q+Bc7, p+ (-1), s+ (+1), and t+u (56) 
Mc8=(0), Bc8 m+ (+1), Bc8, q+Bc8, p+(0). s+ (+1), and t+u (57) 
Mc9 = (+1), Bc9, m+ (+1), Bc9. q+Bc9, p+ (+1). s+ (+1), and t+u (58) 

[0462]When computing the mixture ratio alpha of a pixel contained in a covered background region of frame #n, in a formula (50) thru/or a formula 

(58) , the pixel values Bc1 thru/or Bc9 of a pixel of a background region of a pixel of frame #n-1 corresponding to a pixel of frame #n are used. 
[0463]When computing the mixture ratio alpha of a pixel contained in an uncovered background region shown in drawing 85 , the following formulas 

(59) thru/or f Drmulas (67) is stood, the mixture ratio — a pixel value of a pixel which computes alpha is Mu5. 
[0464] 

Mu1 = (-1), Bui, m+ (-1), Bu1, q+Bu1, p+ (-1), s+ (-1), and t+u (59) 
Mu2=(0), Bu2 m+ (-1), Bu2, q+Bu2. p+(0), s+ (-1), and t+u (60) 
Mu3 = (+1), Bu3, m+ (-1), Bu3, q+Bu3, p+ (+1), s+ (-1), and t+u (61) 
Mu4=(-1), Bui, andm+ (0) -Bu4, q+Bu4 t and p+(-1) -s+(0) and t+u (62) 
Mu5= (0) Bu5, q+Bu5, -Bu5andm+ (0), and p+(0), s+(0), and t+u (63) 
Mu6=(+1). Bu3, andm+ (0) -Bu6, q+Bu6, and p+(+1) -s+(0) and t+u (64) 
Mu7 = (-1), Bu7, m+ (+1), Bu7, q+Bu7, p+ (-1), s+ (+1), and t+u (65) 
Mu8=(0). Bu8 m+ (+1), Bu8, q+Bu8, p+(0), s+ (+1), and t+u (66) 
Mu9 = (+1), Bu9, m+ (+1). Bu9, q+Bu9. p+ (+1). s+ (+1). and t+u (67) 

[0465]When computing the mixture ratio alpha of a pixel contained in an uncovered background region of frame #n, in a formula (59) thru/or a 
formula (67), the pixel values Bu1 thru/or Bu9 of a pixel of a background region of a pixel of frame #n+1 corresponding to a pixel of frame #n are 
used. 

[0466] Drawing 86 is a block diagram showing composition of the presumed mixture ratio treating part 401. A picture inputted into the presumed 
mixture ratio treating part 401 is supplied to the delay part 501 and the help lump part 502. 

[0467]The delay circuit 221 carries out 1 frame delay of the inputted image, is added, and is supplied to the lump part 502. When it adds and frame 
#n is inputted into the lump part 502 as an inputted image, the delay circuit 221 adds frame #n-1 and supplies it to the lump part 502. 
[0468]It adds and the lump part 502 sets a pixel value of a pixe! near the pixel which computes the mixture ratio alpha, and a pixel value of frame 
#n-1 as a normal equation. For example, it adds and the lump part 502 sets the pixel values Mc1 thru/or Mc9 and the pixel values Bc1 thru/or Bc9 



as a normal equation based on an equation (50) thru/or an equation (58). It adds and the lump part 502 supplies a normal equation with which a pixel 
value was set up to the operation part 503. 

[0469]The operation part 503 sweeps out a normal equation which added and was supplied from the lump part 502, solves it by law etc., asks for the 
presumed mixture ratio, and outputs the called-for presumed mixture ratio. 

[0470]Thus, based on an inputted image, the presumed mixture ratio treating part 401 can compute the presumed mixture ratio, and can supply it to 
the mixture rntio deciding part 403. 

[0471]Since i : has the same composition as the presumed mixture ratio treating part 401, the presumed mixture ratio treating part 402 omits the 
explanation. 

[0472] Drawin? 87 is a figure showing an example of the presumed mixture ratio computed by the presumed mixture ratio treating part 401. The 
motion v of a foreground corresponding to an object which is moving at uniform velocity is 1 1, and the presumed mixture ratio shown in drawing 87 
shows a resut computed by having generated an equation to one line by making a 7x7-pixel block into a unit. 
[0473]As the presumed mixture ratio is shown in drawing 86 in a mixing zone, it turns out that it is changing almost linearly. 

[0474]Next, processing of mixture ratio presumption by a model corresponding to a covered background region by the presumed mixture ratio 
treating part 401 which shows drawing 86 composition is explained with reference to a flow chart of drawing 88 . 

[0475]In Step S521, it adds and the lump part 502 sets a pixel value included in an inputted picture, and a pixel value included in a picture supplied 
from the delay circuit 221 as a normal equation corresponding to a model of a covered background region. 

[0476]ln Step S522, when it judges whether setting out about the target pixel was completed and is judged with setting out about the target pixel 
not being completed, the presumed mixture ratio treating part 401 returns to Step S521, and repeats processing of setting out of a pixel value to a 
normal equat on. 

[0477]ln Step S522, when judged with setting out of a pixel value about the target pixel having been completed, it progresses to Step S523, and 
based on a normal equation with which a pixel value was set up, the operation part 173 calculates the presumed mixture ratio, and outputs the 
called-for presumed mixture ratio. 

[0478]Thus, the presumed mixture ratio treating part 401 which shows drawing 86 composition can calculate the presumed mixture ratio based on 
an inputted image. 

[0479]Since processing of mixture ratio presumption by a model corresponding to an uncovered background region is the same as processing shown 
in a flow chait using a normal equation corresponding to a model of an uncovered background region of drawing 88 , the explanation is omitted. 
[0480] Although it explained that an object corresponding to a background was standing it still, processing which asks for the mixture ratio 
mentioned above even if a picture corresponding to a background region included a motion is applicable. For example, while a picture corresponding 
to a background region is moving uniformly, the presumed mixture ratio treating part 401 shifts the whole picture corresponding to this motion, and 
is processed like a case where an object corresponding to a background is standing it still. When a picture corresponding to a background region 
includes a different motion for every part, the presumed mixture ratio treating part 401 chooses a pixel corresponding to a motion as a pixel 
corresponding to a pixel belonging to a mixing zone, and performs above-mentioned processing. 

[0481]thus, the mixture ratio which is the characteristic quantity corresponding to each pixel area information to which the mixture ratio calculation 
part 102 was supplied from the field specific part 101, and based on an inputted image — alpha is computable. 

[0482]the muture ratio — a motion included in a picture corresponding to an object which is moving by using alpha — it becomes possible to 
separate an i igredient of a foreground and an ingredient of a background which are contained in a pixel value, with information on a Japanese quince 
left. 

[0483]a right motion doubled with speed of an object which actually rephotoed the real world, and which is moving when combining a picture based 
on the mixture ratio alpha — it becomes possible to make a picture containing a Japanese quince. 

[0484]Next, the foreground background separation part 105 is explained. Drawing 89 is a block diagram showing an example of composition of the 



foreground background separation part 105. An inputted image supplied to the foreground background separation part 105 is supplied to the 
separation pait 601, the switch 602, and the switch 604. Area information supplied from the field specific part 103 which shows information which 
shows a covered background region, and an uncovered background region is supplied to the separation part 601. Area information which shows a 
foreground region is supplied to the switch 602. Area information which shows a background region is supplied to the switch 604. 
[0485]the mixcure ratio supplied from the mixture ratio calculation part 104 — alpha is supplied to the separation part 601. 

[0486]area in'ormation the separation part 601 indicates a covered background region to be, area information which shows an uncovered 
background region, and the mixture ratio — an ingredient of a foreground from an inputted image based on alpha, [ separate and ] While supplying an 
ingredient of a separated foreground to the synchronizer 603, an ingredient of a background is separated from an inputted image and an ingredient 
of a separated background is supplied to the synchronizer 605. 

[0487]When a pixel corresponding to a foreground is inputted based on area information which shows a foreground region, the switch 602 is closed 
and supplies only a pixel corresponding to a foreground included in an inputted image to the synchronizer 603. 

[0488]When a pixel corresponding to a background is inputted based on area information which shows a background region, the switch 604 is closed 
and supplies only a pixel corresponding to a background included in an inputted image to the synchronizer 605. 

[0489]Based on a pixel corresponding to a foreground supplied from an ingredient corresponding to a foreground supplied from the separation part 
601. and the switch 602, the synchronizer 603 combines a foreground ingredient picture and outputs a compound foreground ingredient picture. 
Since a foreground region and a mixing zone do not overlap, the synchronizer 603 applies an operation of logical sum to an ingredient corresponding 
to a foreground, and a pixel corresponding to a foreground, and combines a foreground ingredient picture, for example. 

[0490]In processing of initialization performed by the beginning of processing of composition of a foreground ingredient picture, all the pixel values 
store a picture which is 0 in a built-in frame memory, and the synchronizer 603 stores a foreground ingredient picture in it in processing of 
composition cf a foreground ingredient picture (overwrite). Therefore, 0 is stored in a pixel corresponding to a background region as a pixel value 
among foreground ingredient pictures which the synchronizer 603 outputs. 

[0491]Based on a pixel corresponding to a background supplied from an ingredient corresponding to a background supplied from the separation part 
601 , and the < witch 604, the synchronizer 605 compounds a background component image and outputs a compound background component image. 
Since a background region and a mixing zone do not overlap, the synchronizer 605 applies an operation of logical sum to an ingredient corresponding 
to a background, and a pixel corresponding to a background, and compounds a background component image, for example. 

[0492]In proc sssing of initialization performed by the beginning of processing of composition of a background component image, all the pixel values 
store a pictu e which is 0 in a built-in frame memory, and the synchronizer 605 stores a background component image in it in processing of 
composition of a background component image (overwrite). Therefore. 0 is stored in a pixel corresponding to a foreground region as a pixel value 
among backgiound component images which the synchronizer 605 outputs. 

[0493] Dra win,; 90 is a figure showing a foreground ingredient picture and a background component image which are outputted from an inputted 
image inputted into the foreground background separation part 105, and the foreground background separation part 105. 

[0494] Drawiq; 90 (A) is a mimetic diagram of a picture displayed, and drawing 90 (B) shows a model figure which developed a pixel of one line 
containing a pixel belonging to a foreground region corresponding to drawing 90 (A), a pixel belonging to a background region, and a pixel belonging to 
a mixing zone to a time direction. 

[0495]As shown in drawing 90 (A) and drawing 90 (B), a background component image outputted from the foreground background separation part 
105 comprises an ingredient of a background included in a pixel belonging to a background region, and a pixel of a mixing zone. 
[0496]As shown in drawing 90 (A) and drawing 90 (B), a foreground ingredient picture outputted from the foreground background separation part 105 
comprises an ingredient of a foreground included in a pixel belonging to a foreground region, and a pixel of a mixing zone. 

[0497]A pixel value of a pixel of a mixing zone is divided into an ingredient of a background, and an ingredient of a foreground by the foreground 
background separation part 105. An ingredient of a separated background constitutes a background component image with a pixel belonging to a 



background region. An ingredient of a separated foreground constitutes a foreground ingredient picture with a pixel belonging to a foreground region. 
[0498]Thus. a pixel value to which a pixel value of a pixel corresponding to a background region in a foreground ingredient picture is set to 0, and a 
pixel correspcnding to a foreground region and a pixel corresponding to a mixing zone have a meaning is set up. Similarly, a pixel value to which a 
pixel value of a pixel corresponding to a foreground region in a background component image is set to 0, and a pixel corresponding to a background 
region and a pixel corresponding to a mixing zone have a meaning is set up. 

[0499]Next, processing which separates an ingredient of a foreground and an ingredient of a background from a pixel belonging to a mixing zone 
which the separation part 601 performs is explained. 

[0500] Drawing 91 is a model of a picture in which an ingredient of a foreground of two frames and an ingredient of a background including a 
foreground ccrresponding to an object which moves to the right from the left in a figure are shown. In a model of a picture shown in drawing 91 , 
movement quantity v of a foreground is 4 and the virtual number of partitions is set to 4. 

[0501 ]In frame #n, leftmost pixel and the left to the 14th thru/or the 18th pixel comprise only an ingredient of a background, and belongs to a 
background r« gion. In frame #n, the 2nd thru/or the 4th pixel belong to an uncovered background region including an ingredient of a background, and 
an ingredient of a foreground from the left. In frame #n, the 11th thru/or the 13th pixel belong to a covered background region including an 
ingredient of ;i background, and an ingredient of a foreground from the left. In frame #n, the left to the 5th thru/or the 10th pixel comprise only an 
ingredient of a foreground, and belongs to a foreground region. 

[0502]ln frama #n+1, the left to the 1st from the left thru/or the 5th pixel, and the 18th pixel comprise only an ingredient of a background, and 
belong to a background region. In frame #n+1, the 6th thru/or the 8th pixel belong to an uncovered background region including an ingredient of a 
background, and an ingredient of a foreground from the left. In frame #n+1, the 15th thru/or the 1 7th pixel belong to a covered background region 
including an ingredient of a background, and an ingredient of a foreground from the left. In frame #n+1, the left to the 9th thru/or the 14th pixel 
comprise only an ingredient of a foreground, and belongs to a foreground region. 

[0503] Drawini; 92 is a figure explaining processing which separates an ingredient of a foreground from a pixel belonging to a covered background 
region. In drawing 92 , alphal thru/or alpha18 are the mixture ratio corresponding to ******** of a pixel in frame #n. In drawing 92 , the 1 5th thru/or 
the 17th pixel belong to a covered background region from the left. 

[0504]The pixel value C15 of the 15th pixel is expressed with a formula (68) from the left of frame #n. 
[0505] 

C15=B15/v+F09/v+F08/v+F07/v =alpha15 B1 5+F09/v+F08/v+F07/v =alpha15 -nd P1 5+F09/v+F08/v+F07/v (68) 

Here, alpha 1!i is the mixture ratio of the 15th pixel from the left of frame #n. P15 is a pixel value of the 15th pixel from the left of frame #n-1. 
[0506]Based on a formula (68), the sum f15 of an ingredient of a foreground of the 1 5th pixel is expressed with a formula (69) from the left of frame 
#n. 

[0507] 

f15=F09/v+F(>8/v+F07/v=C15-alpha15 a ndPl5 (69) 

[0508]Similari/, the sum f16 of an ingredient of a foreground of the 16th pixel is expressed with a formula (70) from the left of frame #n, and the sum 

f 1 7 of an ingredient of a foreground of the 17th pixel is expressed with a formula (71) from the left of frame #n. 

[0509] 

f16=C16-alpha16 «ndP16 (70) 
f17=C17-alpha17 *ndP17 (71) 

[0510]Thus, tie ingredient fc of a foreground included in the pixel value C of a pixel belonging to a covered background region is calculated by a 

formula (72). 

[0511] 

Fc=C-alpha-p(72) 



Pisa pixel value of a pixel to which a frame in front of one corresponds. 

[0512] Drawing 93 is a figure explaining processing which separates an ingredient of a foreground from a pixel belonging to an uncovered background 
region. In drawing 93 , alpha 1 thru/or alpha 18 are the mixture ratio corresponding to ******** of a pixel in frame #n. In drawing 93 . the 2nd thru/or 
the 4th pixel t elong to an uncovered background region from the left. 

[0513]The pixsl value C02 of the 2nd pixel is expressed with a formula (73) from the left of frame #n. 
[0514] 

C02=B02/v+E02/v+B02/v+F01/v=alpha2an<iB02+F01/v=alpha2«ndN02+F01/v (73) 

Here, alpha 2 is the mixture ratio of the 2nd pixel from the left of frame #n. N02 is a pixel value of the 2nd pixel from the left of frame #n+1. 
[0515]Based on a formula (73), the sum f02 of an ingredient of a foreground of the 2nd pixel is expressed with a formula (74) from the left of frame 

#n. 

[0516] 

fO2=F01/v =C02-alpha2 ™* N02 (74) 

[0517]Similarl/, the sum f03 of an ingredient of a foreground of the 3rd pixel is expressed with a formula (75) from the left of frame #n, and the sum 

f04 of an ingredient of a foreground of the 4th pixel is expressed with a formula (76) from the left of frame #n. 

[0518] 

f03=C03-alpha3 «"d N03 (75) 
f04=C04-alpha4 «d N04 (76) 

[0519]Thus, the ingredient fu of a foreground included in the pixel value C of a pixel belonging to an uncovered background region is calculated by a 

formula (77). 

[0520] 

Fu=C-»lpho-N(77) 

N is a pixel vj lue of a pixel to which a frame after one corresponds. 

[0521]Thus, t ie separation part 601 can separate an ingredient of a foreground, and an ingredient of a background from a pixel belonging to a mixing 
zone based on information which shows a covered background region included in area information, information which shows an uncovered 
background region, and the mixture ratio alpha for every pixel. 

[0522] Drawinis 94 is a block diagram showing an example of composition of the separation part 601 which performs processing explained above. The 
mixture ratio alpha is inputted into the separation block 622 at area information and a row which show a covered background region and an 
uncovered background region which a picture inputted into the separation part 601 was supplied to the frame memory 621, and were supplied from 
the mixture nttio calculation part 104. 

[0523]The frame memory 621 memorizes an inputted picture per frame. The frame memory 621 memorizes frame #n+1 which is a frame after [ of 
frame #n-1 which is a frame in front of / of frame #n / one, frame #n and frame #n ] one, when an object of processing is frame #n. 
[0524]The frame memory 621 supplies a pixel to which frame #n-1, frame #n, and frame #n+1 corresponds to the separation block 622. 
[0525]Area information the separation block 622 indicates a covered background region and an uncovered background region to be, An operation 
explained to a pixel value of a pixel to which frame #n-1 supplied to row from frame memory 621 based on the mixture ratio alpha, frame #n t and 
frame #n+1 corresponds with reference to drawing 92 and drawing 93 is applied, An ingredient of a foreground and an ingredient of a background are 
separated froTi a pixel belonging to a mixing zone of frame #n, and the frame memory 623 is supplied. 

[0526]The separation block 622 comprises the uncovered region processing part 631 , the covered region processing part 632, the synchronizer 633, 
and the synchronizer 634. 

[0527]A pixel value of a pixel of frame #n+1 to which the mixture ratio alpha was supplied from the frame memory 621 is multiplied by the multiplier 
641 of the ur covered region processing part 631, and it is outputted to the switch 642. When a pixel (it corresponds to a pixel of frame #n+1) of 



frame #n to which the switch 642 was supplied from the frame memory 621 is an uncovered background region, the mixture ratio which was closed 
and was supplied from the multiplier 641 — a pixel value which multiplied by alpha is supplied to the computing unit 643 and the synchronizer 634. 
A value which multiplied a pixel value of a pixel of frame #n+1 outputted from the switch 642 by the mixture ratio alpha is equal to an ingredient of 
the backgrourd of a pixel value of a pixel that frame #n corresponds. 

[0528]The computing unit 643 subtracts an ingredient of a background supplied from the switch 642 from a pixel value of a pixel of frame #n 
supplied from the frame memory 621, and asks for an ingredient of a foreground. The computing unit 643 supplies an ingredient of a foreground of a 
pixel of frame #n belonging to an uncovered background region to the synchronizer 633. 

[0529]A pixel /alue of a pixel of frame #n-1 to which the mixture ratio alpha was supplied from the frame memory 621 is multiplied by the multiplier 
651 of the covered region processing part 632, and it is outputted to the switch 652. When a pixel (it corresponds to a pixel of frame #n-1 ) of frame 
#n to which the switch 652 was supplied from the frame memory 621 is a covered background region, the mixture ratio which was closed and was 
supplied from the multiplier 651 — a pixel value which multiplied by alpha is supplied to the computing unit 653 and the synchronizer 634. A value 
which multiped a pixel value of a pixel of frame #n-1 outputted from the switch 652 by the mixture ratio alpha is equal to an ingredient of the 
background ol a pixel value of a pixel that frame #n corresponds. 

[0530]The computing unit 653 subtracts an ingredient of a background supplied from the switch 652 from a pixel value of a pixel of frame #n 
supplied from the frame memory 621 , and asks for an ingredient of a foreground. The computing unit 653 supplies an ingredient of a foreground of a 
pixel of frame #n belonging to a covered background region to the synchronizer 633. 

[0531 ]The synchronizer 633 compounds an ingredient of a foreground of a pixel belonging to a covered background region supplied from an 
ingredient of a foreground of a pixel belonging to an uncovered background region supplied from the computing unit 643 of frame #n, and the 
computing un t 653, and supplies it to the frame memory 623. 

[0532], The synchronizer 634 was supplied from the switch 642 of frame #n. An ingredient of the background of a pixel belonging to a covered 
background region supplied from an ingredient of the background of a pixel belonging to an uncovered background region and the switch 652 is 
compounded, and the frame memory 623 is supplied. 

[0533]The frame memory 623 memorizes to each an ingredient of a foreground of a pixel of a mixing zone of frame #n supplied from the separation 
block 622, and an ingredient of a background. 

[0534]The frame memory 623 outputs an ingredient of a foreground of a pixel of a mixing zone of memorized frame #n, and a memorized ingredient 
of the background of a pixel of a mixing zone of frame #n. 

[0535]the mixture ratio which is characteristic quantity — by using alpha, it becomes possible to separate thoroughly an ingredient of a foreground 
and an ingredient of a background which are contained in a pixel value. 

[0536]The sy nchronizer 603 compounds an ingredient of a foreground of a pixel of a mixing zone of frame #n outputted from the separation part 601. 
and a pixel bt longing to a foreground region, and generates a foreground ingredient picture. The synchronizer 605 compounds an ingredient of the 
background cf a pixel of a mixing zone of frame #n outputted from the separation part 601, and a pixel belonging to a background region, and 
generates a kackground component image. 

[0537] Drawing 95 is a figure showing an example of a foreground ingredient picture corresponding to frame #n of drawing 91 . and an example of a 
background component image. 

[0538] Drawing 95 (A) shows an example of a foreground ingredient picture corresponding to frame #n of drawing 91 . From leftmost pixel and the left, 
since the 14tn pixel comprised only an ingredient of a background before a foreground and a background were separated, a pixel value is set to 0. 
[0539]Before a foreground and a background are separated from the left as for the 2nd thru/or the 4th pixel, it belongs to an uncovered background 
region, an ingredient of a background is set to 0, and an ingredient of a foreground is left behind as it is. Before a foreground and a background are 
separated frcm the left as for the 1 1th thru/or the 13th pixel, it belongs to a covered background region, an ingredient of a background is set to 0, 
and an ingredient of a foreground is left behind as it is. From the left, since only an ingredient of a foreground is comprised, the 5th thru/or the 10th 



pixel are left k ehind as it is. 

[0540] Drawin{; 95 (S) shows an example of a background component image corresponding to frame #n of drawing 91 . From leftmost pixel and the 
left, since only an ingredient of a background was comprised before a foreground and a background were separated, the 14th pixel is left behind as 
it is. 

[0541]Before a foreground and a background are separated from the left as for the 2nd thru/ or the 4th pixel, it belongs to an uncovered background 
region, an ingredient of a foreground is set to 0, and an ingredient of a background is left behind as it is. Before a foreground and a background are 
separated from the left as for the 1 1th thru/or the 13th pixel, it belongs to a covered background region, an ingredient of a foreground is set to 0, 
and an ingredient of a background is left behind as it is. From the left, since the 5th thru/or the 10th pixel comprised only an ingredient of a 
foreground before a foreground and a background were separated, a pixel value is set to 0. 

[0542]Next, vnth reference to a flow chart shown in drawing 96 , processing of separation with a foreground and a background by the foreground 
background separation part 105 is explained. In Step S601, the frame memory 621 of the separation part 601 acquires an inputted image, and 
memorizes fr;.me #n which is the target [ background / a foreground and ] of separation with frame #n-1 before that, and frame #n+1 of after that. 
[0543]In Step S602, the separation block 622 of the separation part 601 acquires area information supplied from the mixture ratio calculation part 
104. the mixture ratio to which the separation block 622 of the separation part 601 was supplied from the mixture ratio calculation part 104 in Step 
S603 — alph.i is acquired. 

[0544]in Step S604 — the uncovered region processing part 631 — area information and the mixture ratio — an ingredient of a background is 
extracted from a pixel value of a pixel which belongs based on alpha in an uncovered background region supplied from the frame memory 621. 
[0545]in Step S605 — the uncovered region processing part 631 — area information and the mixture ratio — an ingredient of a foreground is 
extracted from a pixel value of a pixel which belongs based on alpha in an uncovered background region supplied from the frame memory 621. 
[0546]in Step S606 — the covered region processing part 632 — area information and the mixture ratio — an ingredient of a background is 
extracted from a pixel value of a pixel which belongs based on alpha in a covered background region supplied from the frame memory 621. 
[0547]in Step S607 — the covered region processing part 632 ~ area information and the mixture ratio — an ingredient of a foreground is 
extracted fron a pixel value of a pixel which belongs based on alpha in a covered background region supplied from the frame memory 621. 
[0548]In Step S608. the synchronizer 633 compounds an ingredient of a foreground of a pixel belonging to an uncovered background region 
extracted by processing of Step S605, and an ingredient of a foreground of a pixel belonging to a covered background region extracted by 
processing of Step S607. An ingredient of a compounded foreground is supplied to the synchronizer 603. The synchronizer 603 compounds a pixel 
belonging to << foreground region supplied via the switch 602, and an ingredient of a foreground supplied from the separation part 601. and generates 
a foreground ingredient picture. 

[0549]In Step S609. the synchronizer 634 compounds an ingredient of the background of a pixel belonging to an uncovered background region 
extracted by processing of Step S604, and an ingredient of the background of a pixel belonging to a covered background region extracted by 
processing ol Step S606. An ingredient of a compounded background is supplied to the synchronizer 605. The synchronizer 605 compounds a pixel 
belonging to a background region supplied via the switch 604, and an ingredient of a background supplied from the separation part 601, and 
generates a background component image. 

[0550]In Step S610, the synchronizer 603 outputs a foreground ingredient picture. In Step S611. the synchronizer 605 outputs a background 
component inage and ends processing. 

[0551]thus, 1he foreground background separation part 105 — area information and the mixture ratio — based on alpha, an ingredient of a 
foreground and an ingredient of a background can be separated from an inputted image, and a foreground ingredient picture which comprises only an 
ingredient of a foreground, and a background component image which comprises only an ingredient of a background can be outputted. 
[0552]Next, c. foreground ingredient picture moves and adjustment of quantity of a Japanese quince is explained. 

[0553] Drawirg 97 is a block diagram showing an example of composition of the motion dotage controller 106. A motion vector supplied from the 



motion detection part 102 and its position information are supplied to the batch deciding part 801, the modeling part 802, and the operation part 805. 
Area information supplied from the field specific part 103 is supplied to the batch deciding part 801. A foreground ingredient picture supplied from 
the foreground background separation part 105 is added, and is supplied to the lump part 804. 

[0554]Based on a motion vector, its position information, and area information, the batch deciding part 801 generates a batch and supplies a 
generated bat;h to the modeling part 802 and the help lump part 804. 

[0555]A batch which the batch deciding part 801 generates, As an example is shown in drawing 98 , it begins from a pixel corresponding to a covered 
background rugion of a foreground ingredient picture, It begins from a continuous pixel located in a line in the motion direction to a pixel 
corresponding to an uncovered background region, or a pixel corresponding to an uncovered background region, and a continuous pixel located in a 
line in the molion direction to a pixel corresponding to a covered background region is shown. A batch comprises two data, a top left point (position 
of a pixel whic h is a pixel specified by a batch and is located in the leftmost or the top on a picture), and a lower right point, for example. 
[0556]The modeling part 802 performs modeling based on a motion vector and an inputted batch. More specifically, for example the modeling part 
802, Two or rr ore models corresponding to the number of pixels contained in a batch, the virtual number of partitions of a time direction of a pixel 
value, and the number of ingredients of a foreground for every pixel are memorized beforehand, A model which specifies correspondence with a pixel 
value as shown in drawing 99 , and an ingredient of a foreground is chosen based on a batch and the virtual number of partitions of a time direction 
of a pixel value. 

[0557][ for example, / when the number of pixels corresponding to a batch is 12 and movement quantity v within shutter time is 5 ], The modeling 
part 802 sets the virtual number of partitions to 5, and a pixel located in the leftmost contains an ingredient of one foreground, The 2nd pixel 
contains an ir gredient of two foregrounds from the left, and the 3rd pixel contains an ingredient of three foregrounds from the left, The 4th pixel 
contains an ingredient of four foregrounds from the left, and the 5th pixel contains an ingredient of five foregrounds from the left, The 6th pixel 
contains an ingredient of five foregrounds from the left, and the 7th pixel contains an ingredient of five foregrounds from the left. The 8th pixel 
contains an ir gredient of five foregrounds from the left, and the 9th pixel contains an ingredient of four foregrounds from the left, In the left to the 
12th pixel, the left to the 1 1th pixel chooses [ the 10th pixel ] from the left a model which comprises an ingredient of eight foregrounds as a whole 
including an ingredient of one foreground including an ingredient of two foregrounds including an ingredient of three foregrounds. 
[0558]When i1 does not choose from a model memorized beforehand but a motion vector and a batch are supplied, it may be made for the modeling 
part 802 to generate a model based on a motion vector and a batch. 

[0559]The mc deling part 802 supplies a selected model to the equation generation part 803. 

[0560]The eq jation generation part 803 generates an equation based on a model supplied from the modeling part 802. With reference to a model of 
a foreground ngredient picture shown in drawing 99 , the number of ingredients of a foreground is 8, the number of pixels corresponding to a batch 
is 12, movement quantity v is 5 and an equation which the equation generation part 803 in case the virtual number of partitions is 5 generates is 
explained. 

[0561]When foreground ingredients corresponding to the shutter time/v contained in a foreground ingredient picture are F01/v thru/or F08/v, 

relation between F01/v thru/or F08/v, and the pixel values C01 thru/or C12 is expressed with a formula (78) thru/or a formula (89). 

[0562] 

C01=F01/v C8) 
C02=F02/v+F 01/v (79 
C03=F03/v+F02/v+F01/v (80) 
C04=F04/v+F 03/v+F02/v+F0 1 /v (8 1 ) 
C05=F05/v+F 04/v+F03/v+F02/v+F01 /v (82) 
C06=F06/v+F'05/v+F04/v+F03/v+F02/v (83) 
C07=F07/v+F06/v+F05/v+F04/v+F03/v (84) 



C08=F08/v+F')7/v+F06/v+F05/v+F04/v (85) 
C09=F08/v+F07/v+F06/v+F05/v (86) 
C10=F08/v+F)7/v+F06/v (87) 
C11=F08/v+F)7/v (88 
C12=F08/v (89) 

[0563]The equation generation part 803 transforms a generated equation, and generates an equation. An equation which the equation generation 

part 803 gene-ates is shown in an equation (90) thru/or an equation (101). 

[0564] 

F06 [ C01=1 end FOI/v+0 and F02/v+0 and F03/v+0 and F04/v+0 and F05/v+0 and ]/v+0 and F07/v+0 and F08 / v (90) 
F06 [ C02=1 e nd F01/v+1 and F02/v+0 and F03/v+0 and F04/v+0 and F05/v+0 and ]/v+0 and F07/v+0 and F08 / v (91) 
F06 [ C03=1 e nd F01/v+1 and F02/v+1 and F03/v+0 and F04/v+0 and F05/v+0 and ]/v+0 and F07/v+0 and F08 / v (92) 
F06 [ C04=1 e nd F01/v+1 and F02/v+1 and F03/v+1 and F04/v+0 and F05/v+0 and ]/v+0 and F07/v+0 and F08 / v (93) 
F06 [ C05=1 e nd F01/v+1 and F02/v+1 and F03/v+1 and F04/v+1 and F05/v+0 and ]/v+0 and F07/v+0 and F08 / v (94) 
F06 [ C06=0 e nd F01/v+1 and F02/v+1 and F03/v+1 and F04/v+1 and F05/v+1 and ]/v+0 and F07/v+0 and F08 / v (95) 
F06 [ C07=0 e nd F01/V+0 and F02/v+1 and F03/v+1 and F04A/+1 and F05/v+1 and ]/v+1 and F07/v+0 and F08 / v (96) 
F06 [ C08=0 <>nd F01/v+0 and F02/v+0 and F03/v+1 and F04/v+1 and F05/v+1 and ]/v+1 and F07/v+1 and F08 / v (97) 
F06 [ C09=0 c.nd F01/v+0 and F02/v+0 and F03/v+0 and F04/v+1 and F05/v+1 and ]/v+1 and F07/v+1 and F08 / v (98) 
F06 [ C10=0 and F01/v+0 and F02/v+0 and F03/v+0 and F04A/+0 and F05/v+1 and ]/v+1 and F07/v+1 and F08 / v (99) 
F06 [ C11=0 <md F01/v+0 and F02/v+0 and F03/v+0 and F04/v+0 and F05/v+0 and ]/v+1 and F07/v+1 and F08 / v (100) 
F06 [ C12=0 nnd F01/v+0 and F02/V+0 and F03/v+0 and F04/v+0 and F05/v+0 and ]/v+0 and F07/v+1 and F08 / v (101) 
[0565]A formula (90) thru/or a formula (101) can also be expressed as a formula (102). 
[0566] 

[Equation 17] 



In a formula ( 02), j shows the position of a pixel. In this example, j has any one value of 1 thru/or 12. i shows the position of a foreground value. In 
this example, i has any one value of 1 thru/or 8. aij has a value of 0 or 1 corresponding to the value of i and j. 
[0567]If it exp resses in consideration of an error, a formula (102) can be expressed like a formula (103). 
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(102) 



1 = 01 



[0568] 



[Equation 18] 
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In a formula (1 03), ej is an error included in the noticed picture element Cj. 



[0569]A formula (103) can be rewritten at a ceremony (104). 



[0570] 



[Equation 19] 
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i=01 



[0571]Here. in order to apply a least square method, the sum of squares E with error is defined as shown in a formula (105). 



[0572] 

[Equation 20] 

E = f J ej 2 ao5) 

[0573]In order for an error to become the minimum, the value of the partial differential by the variable Fk to the sum of squares E with error should 

just be set to 0. Fk is calculated so that a formula (106) may be filled. 

[0574] 



[Equation 21] 
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[0575]ln a formula (106), since movement quantity v is a fixed value, it can draw a formula (107). 
[0576] 

[Equation 22] 

12 WV 

2 akj • (Cj -Zaij-Fih) = 0 (107) 

[0577]A formula (108) will be obtained if a formula (107) is developed and transposed. 
[0578] 

[Equation 23] 

12 12 

^(akj^aij'Fi) = v 'Zakj* Cj aos) 

i=m i=oi y-« 

[0579]It develops at eight ceremony produced by substituting any one of the integers of 1 thru/or 8 for k of a formula (108). One formula can 
express eight obtained formulas by a procession. This equation is called a normal equation. 

[0580]An example of a normal equation which the equation generation part 803 generates based on such a least square method is shown in an 

equation (109). 

[0581] 

[Equation 24] 
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[0582]lf a formula (109) is expressed as a-f=v-c, C, A, and v are known and F is strange. Although A and v are known at the time of modeling, C 
becomes knov/n in adding and inputting a pixel value in lump operation. 

[0583]The error included in the pixel C can be distributed by computing a foreground ingredient with the normal equation based on a least square 
method. 

[0584]The equation generation part 803 adds a normal equation generated in this way, and supplies it to the lump part 804. 

[0585]It adds and the lump part 804 sets the pixel value C included in a foreground ingredient picture as an equation of a procession supplied from 
the equation feneration par t 803 based on a batch supplied from the batch deciding part 801. It adds and the tump part 804 supplies a procession 
which set up :he pixel value C to the operation part 805. 

[0586]The operation part 805 sweeps out and by processing based on solutions, such as law (elimination of Gauss-Jordan), a motion — computing 
foreground ingredient Fi/v from which a Japanese quince was removed — a motion — R corresponding to one i of the integers of 0 thru/ or 8 which 
is a pixel valus of a foreground where a Japanese quince was removed, [ compute and ] It moves and a foreground ingredient picture which shows 
drawing 100 an example, which comprises Fi which is the pixel value from which it moved and a Japanese quince was removed and from which it 
moved and a Japanese quince was removed is outputted to the Japanese quince adjunct 806 and the selecting part 807. 

[0587]a motion shown in drawing 100 — C03 thru/or C10 are alike, respectively, each of F01 thru/or F08 is set up in order not to change a position 
of a foreground ingredient picture over a screen, and it can be made to correspond to arbitrary positions in a foreground ingredient picture from 
which a Japanese quince was removed 

[0588]a moticn — a Japanese quince — a motion of a value in which the adjunct 806 differs from movement quantity v — a Japanese quince — a 
motion of a v alue of a half of amount of adjustments v', for example, movement quantity v, — a Japanese quince — a motion of amount of 
adjustments x and a value unrelated to movement quantity v — a Japanese quince ~ giving amount of adjustments v' — a motion — quantity of 
a Japanese quince can be adjusted, for example, it is shown in Drawing 101 — as — a motion — a Japanese quince -- the adjunct 806 — a motion 
— moving th<i pixel value Fi of a foreground where a Japanese quince was removed — amount of Japanese quince adjustments v' **** 
computing th<; sum of foreground ingredient Fi/v' by computing foreground ingredient Fi/v' by things — a motion — a pixel value to which quantity 
of a Japanese quince was adjusted is generated. For example, when amount of motion dotage adjustments v' is 3, the pixel value C02 is made into 
(F01) / v\ the pixel value C03 is made into (F01+F02) / v\ the pixel value C04 is made into (F01+F02+F03) / v', and the pixel value COS is made into 
(F02+F03+F04) / v'. 



[0589]The mction dotage adjunct 806 supplies a foreground ingredient picture which moved and adjusted quantity of a Japanese quince to the 
selecting part 807. 

[0590]A foreg-ound ingredient picture from which the selecting part 807 was supplied [ picture ] from the operation part 805, for example based on 
a selection signal corresponding to a user's selection, it moved, and a Japanese quince was removed, And either of the foreground ingredient 
pictures to which were supplied from the motion dotage adjunct 806, it moved, and quantity of a Japanese quince was adjusted is chosen, and a 
selected foreground ingredient picture is outputted. 

[0591]Thus, based on selection signal and amount of motion dotage adjustments v', the motion dotage controller 106 can be moved and can adjust 
quantity of a Japanese quince. 

[0592]For excmple, as shown in Drawing 102 , when the number of pixels corresponding to a batch is 8 and movement quantity v is 4, the motion 
dotage controller 106 generates a formula of a procession shown in a formula (110). 
[0593] 

[Equation 25] 
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[0594]The motion dotage controller 106 computes Fi which is the pixel value to which the formula of the number corresponding to the length of the 
batch was stood in this way, it moved, and the quantity of the Japanese quince was adjusted. When similarly there is the 100 number of the pixels 
contained in ii batch, for example, Fi is computed by generating the formula corresponding to 100 pixels. 

[0595] Drawing 103 is a figure showing other composition of the motion dotage controller 106. The same number is given to the same portion as the 
case where it is shown in drawing 97 , and the explanation is omitted. 

[0596]. [ whether the selecting part 821 supplies an inputted motion vector and its position signal to the batch deciding part 801 and the modeling 
part 802 as ii is based on a selection signal, and ] Or a motion vector which moved, transposed a size of a motion vector to amount of Japanese 
quince adjustments v\ and the size moved and was transposed to amount of Japanese quince adjustments v', and its position signal are supplied to 
the batch deciding part 801 and the modeling part 802. 

[0597]doing h this way — a motion of Drawing 103 — a Japanese quince — the batch deciding part 801 thru/or the operation part 805 of the 
controller 10li moves with movement quantity v, and corresponds to a value with amount of Japanese quince adjustments v' — a motion — quantity 
of a Japanese quince can be adjusted. For example, when amount of Japanese quince adjustments v' is 3, movement quantity v is 5, move, and 
Drawing 103 moves, and the batch deciding part 801 thru/or the operation part 805 of the Japanese quince controller 106, a motion whose 
movement quantity v shown in drawing 99 is 3 to a foreground ingredient picture which is 5 — a Japanese quince — amount of adjustments v' — 
a motion according to a model as shown in corresponding Drawing 101 , perform an operation, and corresponding to (movement quantity v)/(motion 
Japanese quince amount of adjustments v') =5 / 3, i.e., movement quantity v of about 1.7, — a picture containing a Japanese quince is computed, 
a motion corresponding to movement quantity v whose picture computed in this case is 3 — since a Japanese quince is not included — a motion ~ 
a Japanese quince — cautions are required for a point that move with movement quantity v and implications of a relation of amount of Japanese 



quince adjustments v' differ from a result of the adjunct 806. 

[0598]As mentioned above, the motion dotage controller 106 computes a foreground ingredient picture to which a pixel value of a foreground 
ingredient picture was set as a formula which generated and generated a formula, it moved to it corresponding to movement quantity v and a batch, 
and quantity cf a Japanese quince was adjusted. 

[0599]Next, wth reference to a flow chart of Drawing 104 , it moves, and it is contained in a foreground ingredient picture by the Japanese quince 
controller 106 and moves, and processing of adjustment of quantity of a Japanese quince is explained. 

[0600]ln Step S801, it moves and the batch deciding part 801 of the Japanese quince controller 106 supplies a batch which generated and 
generated a batch to the modeling part 802 based on a motion vector and area information. 

[0601 ]In Step S802, it moves and the modeling part 802 of the Japanese quince controller 106 performs selection and generation of a model 
corresponding to movement quantity v and a batch. In Step S803, the equation generation part 803 creates a normal equation based on a selected 
model. 

[0602]In Step S804, it adds and the lump part 804 sets a pixel value of a foreground ingredient picture as a created normal equation. In Step S805, 
add and the lump part 804. When it judges whether a pixel value of all the pixels corresponding to a batch was set up and judged with setting up no 
pixel value of pixels corresponding to a batch, it returns to Step S804 and processing of setting out of a pixel value to a normal equation is repeated. 
[0603]When judged with having set up a pixel value of all the pixels of a batch in Step S805, progress to Step S806 and the operation part 805, A 
pixel value of a foreground which moved based on a normal equation with which a pixel value which added and was supplied from the lump part 804 
was set up, and adjusted quantity of a Japanese quince to it is computed, and processing is ended. 

[0604]Thus, the motion dotage controller 106 can adjust quantity of a foreground-image lost-motion Japanese quince which moves and contains a 
Japanese quince based on a motion vector and area information. 

[0605]That is it can be contained in a pixel value which is sample data, and can move, and quantity of a Japanese quince can be adjusted. 
[0606] Drawint; 105 is a block diagram showing other examples of composition of the motion dotage controller 106. Area information which a motion 
vector supplied from the motion detection part 102 and its position information were supplied to the batch deciding part 901 and the amendment 
part 905, and was supplied from the field specific part 103 is supplied to the batch deciding part 901. A foreground ingredient picture supplied from 
the foreground background separation part 105 is supplied to the operation part 904. 

[0607]The batch deciding part 901 supplies a batch which generated a batch and was generated with a motion vector to the modeling part 902 
based on a motion vector, its position information, and area information. 

[0608]The modeling part 902 performs modeling based on a motion vector and an inputted batch. More specifically, for example the modeling part 
902, Two or more models corresponding to the number of pixels contained in a batch, the virtual number of partitions of a time direction of a pixel 
value, and the number of ingredients of a foreground for every pixel are memorized beforehand, A model which specifies correspondence with a pixel 
value as shown in Drawing 106 , and an ingredient of a foreground is chosen based on a batch and the virtual number of partitions of a time direction 
of a pixel value. 

[0609][ for example, / when the number of pixels corresponding to a batch is 12 and movement quantity v is 5 ]. The modeling part 902 sets the 
virtual number of partitions to 5, and a pixel located in the leftmost contains an ingredient of one foreground, The 2nd pixel contains an ingredient of 
two foregrounds from the left, and the 3rd pixel contains an ingredient of three foregrounds from the left, The 4th pixel contains an ingredient of four 
foregrounds Irom the left, and the 5th pixel contains an ingredient of five foregrounds from the left. The 6th pixel contains an ingredient of five 
foregrounds from the left, and the 7th pixel contains an ingredient of five foregrounds from the left, The 8th pixel contains an ingredient of five 
foregrounds from the left, and the 9th pixel contains an ingredient of four foregrounds from the left. In the left to the 12th pixel, the left to the 1 1th 
pixel chooses [the 10th pixel ] from the left a model which comprises an ingredient of eight foregrounds as a whole including an ingredient of one 
foreground including an ingredient of two foregrounds including an ingredient of three foregrounds. 

[0610]When it does not choose from a model memorized beforehand but a motion vector and a batch are supplied, it may be made for the modeling 



part 902 to generate a model based on a motion vector and a batch. 

[061 1]The equation generation part 903 generates an equation based on a model supplied from the modeling part 902. 

[0612]With reference to a model of a foreground ingredient picture shown in Drawing 106 thru/or Drawing 108 , the number of ingredients of a 
foreground is <l, the number of pixels corresponding to a batch is 12, and an example of an equation which the equation generation part 903 in case 
movement qu.mtity v is 5 generates is explained. 

[0613]When foreground ingredients corresponding to the shutter time/v contained in a foreground ingredient picture are F01/v thru/or F08/v, 
relation between F01/v thru/or F08/v, and the pixel values C01 thru/or C12 is expressed with a formula (78) thru/or a formula (89) as mentioned 
above. 

[0614]When the pixel values C12 and C11 are observed, as the pixel value C12 is shown in a formula (1 11), the pixel value C11 comprises sum of 
products of tre ingredients F08/v of a foreground, and the ingredients F07/v of a foreground only including the ingredients F08/v of a foreground. 
Therefore, it can ask for the ingredients F07/v of a foreground by a formula (112). 
[0615] 

F08/v=C12(111) 
F07/v=C11-C12(112) 

[061 6]lf similarly an ingredient of a foreground included in the pixel values C10 thru/or C01 is taken into consideration, it can ask for the ingredients 

F06/v of a foieground thru/or F01/v by formula (113) thru/or a formula (118). 

[0617] 

F06/v=C10-C11 (113) 
F05/v=C09-C10 (114) 
F04/v=C08-C09 (115) 
F03/v=C07-C08+C12 (116) 
F02/v=C06-C 07+C 11-012(117) 
F01/v=C05-C06+C10-C11 (118) 

[0618]The equation generation part 903 generates an equation for computing an ingredient of a foreground according to a difference of a pixel value 
showing an example at an equation (111) thru/or a ceremony (118). The equation generation part 903 supplies a generated equation to the 
operation par; 904. 

[0619]The operation part 904 computes an ingredient of a foreground based on an equation which set a pixel value of a foreground ingredient 
picture as an equation supplied from the equation generation part 903. and set up a pixel value. The operation part 904 sets the pixel values COS 
thru/or C12 i.s an equation (111) thru/or an equation (118), for example, when an equation (111) thru/or an equation (118) are supplied from the 
equation gent ration part 903. 

[062O]The operation part 904 computes an ingredient of a foreground based on a formula to which a pixel value was set. For example, by the 
operation based on a formula (111) thru/or a formula (118) to which the pixel values C05 thru/or C12 were set, the operation part 904 computes 
the ingrediems F01/v of a foreground thru/or F08/v, as shown in Drawing 107 . The operation part 904 supplies the ingredients F01/v of a 
foreground thru/or F08/v to the amendment part 905. 

[0621 ]The amendment part 905 computes a pixel value of a foreground which multiplied by movement quantity v contained in an ingredient of a 
foreground supplied from the operation part 904 at a motion vector supplied from the batch deciding part 901, moved, and removed a Japanese 
quince. For example, when the ingredients F01/v of a foreground to which the amendment part 905 was supplied from the operation part 904 
thru/or F08// are supplied, By multiplying the ingredients F01/v of a foreground thru/or each of F08/v by movement quantity v which is 5, as 
shown in Drawing 108 , the pixel values F01 thru/or F08 of a foreground which moved and removed a Japanese quince are computed. 
[0622]The amendment part 905 moves and supplies a foreground ingredient picture which was computed as mentioned above and which comprises 



a pixel value of a foreground which moved and removed a Japanese quince to the Japanese quince adjunct 906 and the selecting part 907. 
[0623]a motion — a Japanese quince — a motion of a value in which the adjunct 906 differs from movement quantity v — a Japanese quince — a 
motion of a value of a half of amount of adjustments v', for example, movement quantity v ( — a Japanese quince — a motion of a value unrelated 
to amount of adjustments v' and movement quantity v — a Japanese quince — it is amount of adjustments v' — a motion — quantity of a Japanese 
quince can be adjusted, for example, it is shown in Drawing 101 — as — a motion — a Japanese quince — the adjunct 906 — a motion — moving 
the pixel value Fi of a foreground where a Japanese quince was removed — amount of Japanese quince adjustments v' — **** — computing the 
sum of foreground ingredient Fi/v' by computing foreground ingredient Fi/v' by things — a motion — a pixel value to which quantity of a Japanese 
quince was adjusted is generated. For example, when amount of motion dotage adjustments v' is 3. the pixel value C02 is made into (F01) / v\ the 
pixel value C03 is made into (F01+F02) / v\ the pixel value C04 is made into (F01+F02+F03) / v\ and the pixel value C05 is made into 
(F02+F03+F04) / v\ 

[0624]The motion dotage adjunct 906 supplies a foreground ingredient picture which moved and adjusted quantity of a Japanese quince to the 
selecting part 907. 

[0625]A foreg-ound ingredient picture from which the selecting part 907 was supplied [ picture ] from the amendment part 905, for example based 
on a selectior signal corresponding to a user's selection, it moved, and a Japanese quince was removed, And either of the foreground ingredient 
pictures to which were supplied from the motion dotage adjunct 906. it moved, and quantity of a Japanese quince was adjusted is chosen, and a 
selected foreground ingredient picture is outputted. 

[0626]Thus, based on selection signal and amount of motion dotage adjustments v\ the motion dotage controller 106 can be moved and can adjust 
quantity of a .Japanese quince. 

[0627]Next, a foreground which shows composition in Drawing 105 move and according to the Japanese quince controller 106 moves, and 
processing of adjustment of quantity of a Japanese quince is explained with reference to a flow chart of Drawing 109 . 

[0628]ln Step S901, it moves and the batch deciding part 901 of the Japanese quince controller 106 supplies a batch which generated and 
generated a batch to the modeling part 902 and the amendment part 905 based on a motion vector and area information. 

[0629]In Step S902, it moves and the modeling part 902 of the Japanese quince controller 106 performs selection and generation of a model 
corresponding to movement quantity v and a batch. In Step S903, the equation generation part 903 generates an equation for computing an 
ingredient of si foreground according to a difference of a pixel value of a foreground ingredient picture based on a model chosen or generated. 
[0630]In Step S904. the operation part 904 extracts an ingredient of a foreground from difference of a pixel value based on an equation which set a 
pixel value of a foreground ingredient picture as a created equation and with which a pixel value was set up. In Step S905, when it judges whether an 
ingredient of Jill the foregrounds corresponding to a batch was extracted and is judged with extracting no ingredient of foregrounds corresponding to 
a batch, the cperation part 904 returns to Step S904, and repeats processing of extraction of an ingredient of a foreground. 

[0631 JWhen judged with having extracted an ingredient of all the foregrounds corresponding to a batch in Step S905, progress to Step S906 and the 
amendment p irt 905, The pixel values F01 thru/or F08 of a foreground which amended the ingredients F01 /v of a foreground thru/or each of F08/v 
which was supplied from the operation part 904 based on movement quantity v. moved to it, and removed a Japanese quince are computed. 
[0632]in Step S907 — a motion — a Japanese quince — the adjunct 906 — a motion -- computing a pixel value of a foreground which adjusted 
quantity of a Japanese quince — the selecting part 907 — a motion — a picture or a motion from which a Japanese quince was removed either 
of the pictures to which quantity of a Japanese quince was adjusted is chosen, a selected picture is outputted and processing is ended. 
[0633]Thus, ,i foreground-image lost-motion Japanese quince which is an operation with the easier Japanese quince controller 106 when 
composition is shown in Drawing 105 , and which moves, moves more nearly promptly, and contains a Japanese quince can be adjusted. 
[0634]the conventional motions, such as a Wiener filter, — although an effect is accepted in an ideal state, the technique of removing a Japanese 
quince select vely, a motion which shows composition in Drawing 105 to sufficient effect not being acquired to a actual picture which was quantized 
and contained a noise — a Japanese quince — an accurate motion in which it is quantized and sufficient effect is accepted also in the controller 



106 also to a nctual picture having contained a noise — it becomes removable [ a Japanese quince ]. 

[0635]As mentioned above, the separation server 1 1 which shows drawing 27 composition can be contained in an inputted image, can be moved, and 
can adjust quimtity of a Japanese quince. 

[0636] Prawing 1 10 is a block diagram showing other composition of a function of the separation server 11. 

[0637]The sane number is given to a portion shown in drawing 27 , and same portion, and the explanation is omitted suitably. 

[0638]The field specific part 103 supplies area information to the mixture ratio calculation part 104 and the synchronizer 1001. 

[0639]The muture ratio calculation part 104 supplies the mixture ratio alpha to the foreground background separation part 105 and the 

synchronizer 001. 

[0640]The forsground background separation part 105 supplies a foreground ingredient picture to the synchronizer 1001. 

[0641]The synchronizer 1001 outputs image composing which combined arbitrary background images and a foreground ingredient picture supplied 
from the foreground background separation part 105 based on area information supplied from the mixture ratio alpha supplied from the mixture ratio 
calculation pa~t 104, and the field specific part 103 and in which arbitrary background images and a foreground ingredient picture were combined. 
[0642] Drawing 1 1 1 is a figure showing composition of the synchronizer 1001. Based on the mixture ratio alpha and arbitrary background images, the 
background component generation part 1021 generates a background component image, and supplies it to the mixing zone image synthesis section 
1022. 

[0643]By corr bining a background component image and a foreground ingredient picture which were supplied from the background component 
generation pai; 1021, the mixing zone image synthesis section 1022 generates mixing zone image composing, and supplies generated mixing zone 
image compos ing to the image synthesis section 1023. 

[0644]The imnge synthesis section 1023 compounds a foreground ingredient picture, mixing zone image composing supplied from the mixing zone 
image synthesis section 1022, and arbitrary background images based on area information, and generates and outputs image composing. 
[0645]Thus, the synchronizer 1001 can combine a foreground ingredient picture to arbitrary background images. 

[0646]the mixture ratio which is characteristic quantity — a picture acquired by combining a foreground ingredient picture with arbitrary 
background images based on alpha changes with a more natural thing as compared with a picture which only compounded a pixel. 
[0647] Drawin^ 1 12 is a block diagram showing composition of further others of a function of the separation server 1 1 which adjusts quantity of 
motion dotage. The separation server 11 shown in Drawing 112 performs calculation of field specification and the mixture ratio alpha in parallel to 
the separation server 1 1 shown in drawing 27 performing calculation of field specification and the mixture ratio alpha in order. 
[0648]The same number is given to the same portion as a function shown in a block diagram of drawing 27 , and the explanation is omitted. 
[0649]An inputted image is supplied to the mixture ratio calculation part 1101, the foreground background separation part 1102, the field specific 
part 103, and the object extraction part 101. 

[0650]The presumed mixture ratio when the mixture ratio calculation part 1 101 assumes that a pixel belongs to a covered background region based 
on an inputted image, And the presumed mixture ratio at the time of assuming that a pixel belongs to an uncovered background region, The 
presumed mixture ratio at the time of assuming that it computes to each of a pixel contained in an inputted image, and a computed pixel belongs to 
a covered background region, And the presumed mixture ratio at the time of assuming that a pixel belongs to an uncovered background region is 
supplied to the foreground background separation part 1 102. 

[0651] Drawin; 1 13 is a block diagram showing an example of composition of the mixture ratio calculation part 1 101. 

[0652]The presumed mixture ratio treating part 401 shown in Drawing 113 is the same as the presumed mixture ratio treating part 401 shown in 
drawing 72 . Tie presumed mixture ratio treating part 402 shown in Drawing 1 13 is the same as the presumed mixture ratio treating part 402 shown 
in drawing 72 

[0653]Based on an inputted image, by the operation corresponding to a model of a covered background region, the presumed mixture ratio treating 
part 401 comoutes the presumed mixture ratio for every pixel, and outputs the computed presumed mixture ratio. 



[0654]Based on an inputted image, by the operation corresponding to a model of an uncovered background region, the presumed mixture ratio 
treating part 402 computes the presumed mixture ratio for every pixel, and outputs the computed presumed mixture ratio. 

[0655]. The foreground background separation part 1 102 was supplied from the mixture ratio calculation part 1101. The presumed mixture ratio at 
the time of assuming that a pixel belongs to a covered background region, And based on area information supplied from the presumed mixture ratio 
at the time of assuming that a pixel belongs to an uncovered background region, and the field specific part 103, a foreground ingredient picture is 
generated from an inputted image, it moves and a generated foreground ingredient picture is supplied to the Japanese quince controller 106 and the 
selecting part 107. 

[0656] Drawin§ 1 14 is a block diagram showing an example of composition of the foreground background separation part 1 102. 

[0657]The sane number is given to the same portion as the foreground background separation part 105 shown in drawing 89 , and the explanation is 
omitted. 

[0658]The selecting part 1121 based on area information supplied from the field specific part 103, The presumed mixture ratio at the time of 
assuming that a pixel belongs to a covered background region supplied from the mixture ratio calculation part 1101, And either of the presumed 
mixture ratio at the time of assuming that a pixel belongs to an uncovered background region is chosen, and the separation part 601 is supplied by 
making the selected presumed mixture ratio into the mixture ratio alpha. 

[0659]Based on the mixture ratio alpha supplied from the selecting part 1 121, and area information, the separation part 601 extracts an ingredient 
of a foreground, and an ingredient of a background from a pixel value of a pixel belonging to a mixing zone, and supplies an ingredient of an extracted 
foreground to the synchronizer 603, and it supplies an ingredient of a background to the synchronizer 605. 
[0660]The separation part 601 can be considered as composition shown in drawing 94 , and the same composition. 

[0661]The synchronizer 603 combines and outputs a foreground ingredient picture. The synchronizer 605 compounds and outputs a background 
component irr age. 

[0662]a moticn shown in Drawing 112 — a Japanese quince — a motion included in a foreground ingredient picture which could consider the 
controller 106 as the same composition as a case where it is shown in drawing 27 , and was supplied from the foreground background separation part 
1 102 based on area information and a motion vector — adjusting quantity of a Japanese quince — a motion — a foreground ingredient picture to 
which quantity of a Japanese quince was adjusted is outputted. 

[0663]The selecting part 107 shown in Drawing 112 , for example based on a selection signal corresponding to a user's selection, Either one of a 
foreground ingredient picture supplied from the foreground background separation part 1102 and a foreground ingredient picture to which was 
supplied from the motion dotage controller 106, it moved, and quantity of a Japanese quince was adjusted is chosen, and a selected foreground 
ingredient pic :ure is outputted. 

[0664]Thus, to a picture corresponding to an object of a foreground included in an inputted image, the separation server 11 which shows 
composition in Drawing 112 can be contained in the picture, can be moved, and can adjust and output quantity of a Japanese quince, the mixture 
ratio which is the buried information like [ the separation server 1 1 which shows composition in Drawing 112 ] the 1st example — the mixture ratio 
which computed and computed alpha — alpha can be outputted. 

[0665] Drawin); 1 1 5 is a block diagram showing other composition of a function of the separation server 11 which combines a foreground ingredient 
picture with arbitrary background images. The separation server 1 1 shown in Drawing 1 1 5 performs calculation of field specification and the mixture 
ratio alpha pa'allel to the separation server 1 1 shown in Drawing 1 1 0 performing calculation of field specification and the mixture ratio alpha serially. 
[0666]The same number is given to the same portion as a function shown in a block diagram of Drawing 112 , and the explanation is omitted. 
[0667]The presumed mixture ratio when the mixture ratio calculation part 1 101 shown in Drawing 115 assumes that a pixel belongs to a covered 
background r-igion based on an inputted image, And the presumed mixture ratio at the time of assuming that a pixel belongs to an uncovered 
background rogion, The presumed mixture ratio at the time of assuming that it computes to each of a pixel contained in an inputted image, and a 
computed pixel belongs to a covered background region, And the presumed mixture ratio at the time of assuming that a pixel belongs to an 



uncovered bac kground region is supplied to the foreground background separation part 1102 and the synchronizer 1201. 

[0668]. The fcreground background separation part 1102 shown in Drawing 115 was supplied from the mixture ratio calculation part 1101. The 
presumed mixt ure ratio at the time of assuming that a pixel belongs to a covered background region, And based on area information supplied from 
the presumed mixture ratio at the time of assuming that a pixel belongs to an uncovered background region, and the field specific part 103, a 
foreground ingredient picture is generated from an inputted image, and a generated foreground ingredient picture is supplied to the synchronizer 
1201. 

[0669]The presumed mixture ratio at the time of assuming that a pixel belongs to a covered background region to which the synchronizer 1201 was 
supplied from the mixture ratio calculation part 1101, And background images arbitrary based on area information supplied from the presumed 
mixture ratio et the time of assuming that a pixel belongs to an uncovered background region, and the field specific part 103. Image composing which 
combined a foreground ingredient picture supplied from the foreground background separation part 1102 and in which arbitrary background images 
and a foreground ingredient picture were combined is outputted. 

[0670] Drawins, 1 16 is a figure showing composition of the synchronizer 1201. The same number is given to the same portion as a function shown in 
a block diagra-n of Drawing 111 , and the explanation is omitted. 

[0671]The selecting part 1221 based on area information supplied from the field specific part 103, The presumed mixture ratio at the time of 
assuming thai a pixel belongs to a covered background region supplied from the mixture ratio calculation part 1101, And either of the presumed 
mixture ratio ;it the time of assuming that a pixel belongs to an uncovered background region is chosen, and the background component generation 
part 1021 is supplied by making the selected presumed mixture ratio into the mixture ratio alpha. 

[0672]Based an the mixture ratio alpha supplied from the selecting part 1221, and arbitrary background images, the background component 
generation pa~t 1021 shown in Drawing 116 generates a background component image, and supplies it to the mixing zone image synthesis section 
1022. 

[0673]By combining a background component image and a foreground ingredient picture which were supplied from the background component 
generation part 1021, the mixing zone image synthesis section 1022 shown in Drawing 1 1 6 generates mixing zone image composing, and supplies 
generated mi>ing zone image composing to the image synthesis section 1023. 

[0674]The imige synthesis section 1023 compounds a foreground ingredient picture, mixing zone image composing supplied from the mixing zone 
image synthesis section 1022, and arbitrary background images based on area information, and generates and outputs image composing. 
[0675]Thus, tne synchronizer 1201 can combine a foreground ingredient picture to arbitrary background images. 

[0676]AlthouKh an ingredient of a background included in a pixel value carried out the mixture ratio alpha comparatively and it was explained, it is 
good also as j rate of an ingredient of a foreground included in a pixel value. 

[06 7 7] Although the direction of a motion of an object used as a foreground was explained as the right from the left, of course, it is not limited in the 
direction. 

[0678]Althoujfh a case where projection of a between [ space-time which has two-dimensional space and time base information for a picture of 
three-dimensional space and real space which has time base information using a video camera ] was performed above was made into an example, 
When amend distortion generated by that projection when the 1st information on more 1st dimension but not only this example is projected on the 
2nd information on the 2nd fewer dimension, extracting significant information or combining a picture with nature more, can be adapted of this 
invention. 

[0679]. The j ensors 76a are ****** and a solid state image pickup device at CCD. For example, BBD (Bucket Brigade Device), CID (Charge 
Injection Device), CPD (Charge Priming Device), Or a CMOS (Complementary Mental Oxide Semiconductor) sensor may be used, and a sensor 
located in a line with one row may be sufficient not only as a sensor by which a sensing element is arranged at matrix form but a sensing element. 
[0680]Each function of the separation server 1 1 explained as mentioned above can also be realized by carrying out distributed processing to various 
kinds of servers constituted on the network 1 shown by drawing 1 . Namely, the object extraction part 101 and the motion detection part 102 to the 



motion detection server 12. The field specific part 103 moves to the field specific server 13, the mixture ratio calculation part 104 moves to the 
mixture ratio calculation server 14, the foreground background separation processing part 105 moves to the foreground background separation 
processing se ver 15, and the Japanese quince controller 106 moves and functions as a thing respectively corresponding to the Japanese quince 
adjustment server 16. Therefore, block diagrams of the separation server 1 1 shown by drawing 27 may be any although what is realized by hardware, 
a thing realized by software, or the network 1 realized. The same may be said of the synthetic server 19, and the composition may be any although 
what correspcnds to the synchronizer 1201 and was realized by hardware, a thing realized by software, or the network 1 realized. 
[0681]the above object extraction part 101, the motion detection part 102. the field specific part 103, the mixture ratio calculation part 104, and the 
foreground background separation processing part 105 — and, Since each processing of the motion dotage controller 106 is the same as that of the 
motion detection server 12, the field specific server 13, the mixture ratio calculation server 14, the foreground background separation processing 
server 15, an J a thing that moved and replaced the Japanese quince adjustment server 16 and the same may be said of the processing, the 
explanation is omitted. 

[0682]The separation server 1 1 as mentioned above Hardware, Or when realizing as software, it is good also as composition included in each of 
various kinds of servers connected to drawing 1 on the network 1 of a statement, the client computer 27, and the camera terminal device 28 as 
separation sections. Then, in the following explanation, when the separation server 1 1 is explained as a single device which has the function to only 
divide an inpu Aed picture into a foreground ingredient picture and a background component image, it shall also be called the separation sections 1 1 . 
[0683]Next, with reference to a flow chart of Drawing 1 1 7 , processing of separation service of a picture by the separation server 1 1 in which it was 
inputted from the client computer 27 via the network 1 of drawing 1 is explained. 

[0684]In Step S1001, the client computer 27 outputs information which specifies a picture to the separation server 11. That is, picture ID which is 
a concrete picture or specifies a picture as information which specifies a picture which a user wants to separate is outputted to the separation 
server 1 1 . 

[0685]In Step SI 01 1 , the separation server 1 1 acquires a specified picture. That is, when information which specifies the picture when a picture has 
been transmitted from the client computer 27 has been transmitted, a picture corresponding to the picture ID is read and acquired from on the 
network 1. 

[0686]In Step S1 01 2, the accounting part 1 1 a of the separation server 1 1 performs accounting with the fee collection server 24 via the network 1 . 
In Step S102 , the fee collection server 24 performs accounting with the separation server 1 1 simultaneously. 

[0687]Here, above-mentioned accounting is explained with reference to a flow chart of Drawing 1 1 8 . Although actual accounting is performed by 
the separation server 1 1 and the fee collection server 24, since it is outputted also from the client computer 27, also about processing of the client 
computer 27, it doubles and information required for various kinds of processings is explained here. 

[0688]As sho/vn in Drawing 119 , in Step S1 101 the client computer 27, Rental spending is transmitted to the separation server 11 via the network 
1 with ID info-mation and certification information (password etc.) which specify service and identify a user (user who receives offer of separation 
service of a picture). That is, when transmitting information which specifies a picture by processing of Step S1001 of Drawing 1 17 in now, 
processing of this step S1 101 will be performed. Rental spending is a fee concerning separation service. 

[0689]In Step SI 111, as shown in Drawing 119 , the accounting part 11a of the separation server 11 receives ID information and certification 
information, and transmits ID of rental spending and oneself to the fee collection server 24 further. 

[0690]In Step S1 121, as shown in Drawing 119 , the fee collection server 24 is asked to the financial server 25 in which a financial institution of a 
customer account manages certification information, customer account ID, and rental spending based on ID transmitted from the separation server 
11. 

[0691]ln Step S1 131, as shown in Drawing 119 , based on customer account ID and certification information, the financial server (for customers) 25 
performs authenticating processing, and notifies an authentication result and information on propriety of use to the fee collection server 24. 
[0692]In Step S1 122, as shown in Drawing 119 , the fee collection server 24 transmits an authentication result and information on use propriety to 



the separatior server 11. In the following explanation, an authentication result advances explanation under conditions that the use is possible, 
satisfactorily. There is a problem in an authentication result, and the processing will be ended when information that the use is not accepted is 
received. 

[0693]In Step S11 12, as shown in Drawing 119 , the separation server 11 does not have a problem in an authentication result, and when it is the 
conditions that use of a financial institution is possible, it provides service to the client computer 27. In Step S1 102, the client computer 27 receives 
offer of service. Namely, in now, in Step S1 112 the separation server 1 1, A specified picture is divided into a foreground ingredient picture and a 
background component image, it outputs to the client computer 27, and the client computer 27 receives a foreground ingredient picture and a 
background component image which were separated in Step S1102. 

[0694]In Step S1 1 13, the separation server 1 1 transmits a notice of use of service to the fee collection server 24. In Step S1 123, the fee collection 
server 24 notifies customer account ID, rental spending, and donor account ID to the financial server (for customers) 25. 

[0695]In Step S1 1 32. the financial server (for customers) 25 transfers rental spending to the donor financial server (for donors) 26 from an account 
of customer account ID. 

[0696]Here, it returns to explanation of a flow chart of Drawing 117 . 

[0697]ln Step S1012 and S1021, after accounting is performed between the separation server 11 and the fee collection servers 24, in Step S1013, 
the separation server 11 performs separation of a picture. Namely, the field specific part 103 of the separation server 11, Processing of field 
specification in which it explained with a flow chart of drawing 53 the mixture ratio calculation part 104, With reference to a flow chart of drawing 81, 
processing of explained calculation of the mixture ratio the foreground background separation part 105, Regulated treatment of motion blur quantity 
the Japanese quince controller 106 explained processing of separation with a foreground and a background of having explained with reference to a 
flow chart of drawing 96 to be with a flow chart of drawing 44 by moving is performed, respectively, and a specified picture is separated. Since each 
of regulated treatment of motion blur quantity, processings of field specification, processings of calculation of the mixture ratio, and processings of 
separation with a foreground and a background is the same as that of ****, the explanation is omitted. 

[0698]In Step S1014, the separation server 11 gives ID to a separated foreground ingredient picture and a background component image, and 
transmits to the client computer 27. In Step S1002. receive a foreground ingredient picture and a background component image which have been 
transmitted from the separation server 1 1 and which are separated, and its ID, and it memorizes to its storage parts store 48 ( drawing 2 ), and the 
client computer 27 is printed out if needed. A foreground ingredient picture and a background component image in which separation of the client 
computer 27 was carried out by the separation server 1 1 according to a user's instructions, Separation server 1 1 self is made to memorize, or it can 
be made to memorize by making it output to the storage servers 18 via the network 1 (accumulation). 

[0699]Althou*;h processing at the time of saying a fee concerning separation transferred by the fee collection server 24 to the financial servers 25 
and 26 in the above explanation has been explained, For example, whenever it makes the storage parts store 48 ( drawing 2 ) memorize a point which 
shows that a -jser paid a donor of separation service a utilization charge beforehand like a prepaid point and receives offer of separation service, as 
the point is subtracted, it may be made to perform accounting. 

[0700]Here, with reference to a flow chart of Drawing 120 , accounting at the time of using a prepaid point is explained. 

[0701 ]In Step S1201, the client computer 27 specifies service and transmits ID information and certification information. That is, the client 
computer 27 performs processing in Step S1101 of Drawing 117 , and same processing. 

[0702]ln Step SI 21 1 , the accounting part 11a of the separation server 1 1 receives ID information and certification information. In Step S1212. the 
accounting pnrt 11a subtracts and memorizes a point equivalent to rental spending concerning separation from a prepaid point equivalent to the 
amount of money beforehand paid in by user of the client computer 27 memorized by the storage parts store 48. In Step SI 21 3. the separation 
server 11 provides service. That is, in now, the separation server 11 performs separation of an inputted picture, and transmits a foreground 
ingredient picture and a background component image which were separated to the client computer 27. 

[0703]In Steo S1202, the client computer 27 receives offer of service. That is, in now, the client computer 27 receives a foreground ingredient 



picture and a background component image which are transmitted from the separation server 11. 

[0704]Althougi a case where the separation server 1 1 has memorized a prepaid point above at its storage parts store 48 ( drawing 2 ) has been 
explained, it becomes processing with the same said of a case where a card with which a prepaid point was recorded, and what is called a prepaid 
card are used for example. In this case, the client computer 27 reads a prepaid point memorized by prepaid card at Step S1 201 , It transmits, and the 
separation server 11 needs to transmit a point which subtracts a point equivalent to a utilization charge from a point received by accounting, and 
brings a subtraction result to the client computer 27, and needs to overwrite a prepaid card. 

[0705]Next, with reference to a flow chart of Drawing 121 , a motion vector of a picture specified from the client computer 27 and processing of 
motion detection service in which position information is searched for are explained. 

[0706]In Step S1301, the client computer 27 outputs information which specifies a picture to the motion detection server 12. That is, picture ID 
which is a concrete picture or specifies a picture as information which specifies a picture a user wants to carry out motion detection processing is 
outputted to the motion detection server 12. 

[0707]In Step S1311, the motion detection server 12 acquires a specified picture. That is, when information which specifies the picture when a 
picture has been transmitted from the client computer 27 has been transmitted, a picture corresponding to the picture ID is read and acquired from 
on the network 1 . 

[0708]In Step S1312 and S1321. the accounting part 12c of the motion detection server 12 and the fee collection server 24 perform accounting. 
About accounting, in Drawing 118 and Drawing 120 . since it is the same processing as a case in separation service, the explanation is omitted. 
[0709]In Step S1313, each object is extracted, the motion detection part 12b detects position information and a motion vector from an acquired 
picture which was specified, and the object extraction part 12a of the motion detection server 12 transmits to the client computer 27. 
[0710]In Step S1302, the client computer 27 receives and memorizes position information and a motion vector of an object which have been 
transmitted from the motion detection server 12. 

[071 1]Position information and a motion vector from which the client computer 27 was detected by the motion detection server 12 according to a 
user's instructions, Motion detection server 1 2 self is made to memorize, or it can be made to memorize by making it output to the storage servers 
18 via the network 1 (accumulation). 

[0712]Next, with reference to a flow chart of Drawing 122 , processing of field specific service in which a field is pinpointed is explained from 
information wiich is performed by the field specific server 13 and which specifies a picture inputted from the client computer 27. and an object. 
[0713]In Step S1401, the client computer 27 outputs information which specifies a picture and an object to the field specific server 13. That is, 
information wiich is a picture concrete as information which specifies a picture a user wants to carry out field specification, or specifies an object 
with picture II) which specifies a picture is outputted to the field specific server 13. 

[0714]In Step S1411, the field specific server 13 acquires a specified picture. That is, when picture ID which specifies the picture when a picture 
has been transmitted from the client computer 27 has been transmitted, a picture corresponding to the picture ID is read and acquired from on the 
network 1. 

[0715]In Step S1412 and S1421, the accounting part 13a of the field specific server 13 and the fee collection server 24 perform accounting. About 
accounting, ir Drawing 118 and Drawing 120 , since it is the same processing as a case in separation service, the explanation is omitted. 
[0716]In Step S1413, the field specific server 13 performs processing of field specification based on information which specifies an object. Since 
processing of field specification is the same as processing explained with reference to a flow chart of drawing 53 , the explanation is omitted. 
[0717]In Step S1414, the field specific server 13 transmits area information required in processing of Step S1413 to the client computer 27. 
[0718]In Step S1402. the client computer 27 receives and memorizes area information transmitted from the field specific server 13. 
[071 9]Field specific server 13 self is made to memorize the client computer 27, or area information searched for by the field specific server 13 is 
made to outpjt to the storage servers 18 via the network 1, and it can make it memorize according to a user's instructions (accumulation). 
[0720]next, computing the mixture ratio with reference to a flow chart of Drawing 123 from information which is performed by the mixture ratio 



calculation server 14 and which specifies a picture inputted from the client computer 27, and an object, and area information — the mixture ratio 
processing of calculation service is explained. 

[0721]In Step S1501, the client computer 27 outputs information which specifies a picture and an object, and area information to the mixture ratio 
calculation seiver 14. That is, picture ID which is a picture concrete as information which specifies a picture a user wants to compute the mixture 
ratio, or specifies a picture, information which specifies an object, and area information are outputted to the mixture ratio calculation server 14. 
[0722]In Step S 151 1, the mixture ratio calculation server 14 acquires a specified picture. That is, when picture ID which specifies the picture when 
a picture has :>een transmitted from the client computer 27 has been transmitted, a picture corresponding to the picture ID is read and acquired 
from on the network 1. 

[0723]In Step S1512 and S1521, the accounting part 14a of the mixture ratio calculation server 14 and the fee collection server 24 perform 
accounting. About accounting, in Drawing 118 and Drawing 120 , since it is the same processing as a case in separation service, the explanation is 
omitted. 

[0724]In Step S1513, the mixture ratio calculation server 14 performs processing of calculation of the mixture ratio based on information which 
specifies an object, and area information. Since processing of calculation of the mixture ratio is the same as processing explained with reference to 
a flow chart o r drawing 81 , the explanation is omitted. 

[0725]In Step S1514, the mixture ratio calculation server 14 transmits the mixture ratio required in processing of Step S1513 to the client computer 
27. 

[0726]In Step S1502, the client computer 27 receives and memorizes the mixture ratio transmitted from the mixture ratio calculation server 14. 
[0727]Mixture ratio calculation server 14 self is made to memorize the client computer 27, or the mixture ratio called for by the mixture ratio 
calculation se ver 1 4 is made to output to the storage servers 1 8 via the network 1 , and it can make it memorize according to a user's instructions 
(accumulation). 

[0728]. Next, the foreground background separation server 15 performs with reference to a flow chart of Drawing 124 . Processing of service 
separated into a foreground ingredient picture and a background component image from information, area information, and the mixture ratio which 
specify a picture inputted from the client computer 27 and an object is explained. 

[0729]In Step S1 601, the client computer 27 outputs information, area information, and information on the mixture ratio that a picture and an object 
are specified :o the foreground background separation server 15. That is, picture ID which is a picture concrete as information which specifies a 
picture a user wants to carry out foreground background separation, or specifies a picture, information which specifies an object, area information, 
and information on the mixture ratio are outputted to the foreground background separation server 15. 

[0730]In Step S1611. the foreground background separation server 15 acquires a specified picture. That is, when picture ID which specifies the 
picture when ,i picture has been transmitted from the client computer 27 has been transmitted, a picture corresponding to the picture ID is read and 
acquired from on the network 1 . 

[0731 ]In Step S1612 and S1621, the accounting part 15a of the foreground background separation server 15 and the fee collection server 24 
perform accounting. About accounting, in Drawing 118 and Drawing 120 , since it is the same processing as a case in separation service, the 
explanation is omitted. 

[0732]In Step SI 61 3, the foreground background separation server 15 performs processing of separation of a foreground and a background based 
on information, area information, and the mixture ratio which specify an object. Since processing of separation of a foreground and a background is 
the same as orocessing explained with reference to a flow chart of drawing 96 , the explanation is omitted. 

[0733]In Step S1614, the foreground background separation server 15 gives ID to a foreground ingredient picture and a background component 
image required in processing of Step S1613, and transmits to the client computer 27. 

[0734]ln Step S1602, the client computer 27 receives and memorizes a foreground ingredient picture and a background component image which 
have been transmitted from the foreground background separation server 15. 



[0735]A foreground ingredient picture and a background component image in which the client computer 27 has been transmitted by the foreground 
background separation server 15 according to a user's instructions, Foreground background separation server 15 self is made to memorize, or it can 
be made to memorize by making it output to the storage servers 18 via the network 1 (accumulation). 

[0736]. Next, vith reference to a flow chart of Drawing 125 , move and the Japanese quince adjustment server 16 performs. Information and a 
motion vector which specify a picture inputted from the client computer 27, and processing of service in which move, and a specified picture moves 
and a Japanese quince is adjusted from the amount of Japanese quince adjustments are explained. 

[07 3 7] inform a- .ion as which the client computer 27 specifies a picture in Step S1701, a motion vector, and a motion — a Japanese quince it 
moves and information on the amount of adjustments is outputted to the Japanese quince adjustment server 16. namely, picture ID which is a 
picture concrete as information which specifies a picture which move, and a user wants to fade and to adjust, or specifies a picture, information 
which specifies an object, a motion vector, and a motion — a Japanese quince ~ information on the amount of adjustments moves and it is 
outputted to the Japanese quince adjustment server 16. 

[0738]In Step S1 71 1 , it moves and the Japanese quince adjustment server 1 6 acquires a specified picture. That is, when picture ID which specifies 
the picture when a picture has been transmitted from the client computer 27 has been transmitted, a picture corresponding to the picture ID is read 
and acquired from on the network 1. 

[0739]In Step S1712 and SI 721 . it moves and the accounting part 16a of the Japanese quince adjustment server 16 and the fee collection server 
24 perform ac counting. About accounting, in Drawing 118 and Drawing 120 , since it is the same processing as a case in separation service, the 
explanation is omitted. 

[0740]in Step S1 71 3 — a motion — a Japanese quince — the adjustment server 1 6 — a motion vector and a motion — a Japanese quince — being 
based on information on the amount of adjustments — a motion — processing of adjustment of quantity of a Japanese quince is performed. Since 
processing of adjustment of quantity of motion dotage is the same as processing explained with reference to a flow chart of Drawing 104, the 
explanation is omitted. 

[0741 ]In Step S1714. it moves, and the Japanese quince adjustment server 16 was called for. moves by processing of Step S1713, gives ID to a 
Japanese quince adjustment picture, and transmits to the client computer 27. 

[0742]ln Step S1702, the client computer 27 moves, it has been transmitted from the Japanese quince adjustment server 16. and it moves, and 
receives and Tiemorizes a Japanese quince adjustment picture. 

[0743]The client computer 27 moves, and a Japanese quince adjustment picture is made to output to the storage servers 18, and it can make it to 
have been transmitted by the Japanese quince adjustment server 16, to move, to make motion dotage adjustment server 16 self memorize, or 
memorize according to a user's instructions (accumulation). 

[0744]Next, detailed composition of the coding server 17 is explained with reference to Drawing 126 . A picture as which the separation sections 
2002 of the coding server 1 7 were inputted (picture ID which specifies a picture is inputted and a corresponding picture via the network 1 ) a picture 
read from the storage servers 1 8 — containing — it separates into a foreground ingredient picture and a background component image, and outputs 
to the coding part 2001 with the mixture ratio, a motion vector, and position information. The separation sections 2002 are things like the separation 
server (separation sections) 1 1 explained with reference to drawing 27 , and since the same may be said of the mixture ratio, a motion vector, and 
acquisition processing of position information, they omit the explanation. 

[0745]The coding part 2001 makes the storage servers 18 output and memorize a foreground ingredient picture and a background component image 
which are inp itted from the separation sections 2002 via the network 1, and. It changes into information, including position information on a network 
of the storags servers 18 made to memorize, i.e.. URL etc., and outputs as foreground ingredient image position data and background component 
image positioi information. At this time, the coding part 2001 outputs also about the mixture ratio, a motion vector, and position information which 
are extractec when separating a foreground ingredient picture and a background component image. 

[0746]When i foreground ingredient picture and a background component image are changed into foreground ingredient image position data and 



background component image position information by the coding part 2001, respectively, the accounting part 17a ( drawing 16 , drawing 1 7 ) performs 
accounting to the fee collection server 24 via the network 1. The synthetic server 19 mentioned later is used for this accounting, and a user who 
receives synthetic service which generates image composing may be made to pay it. When a user using coding service pays beforehand conversely 
and synthetic service is used, it can be necessary to receive payment of a utilization charge from a user. 

[0747]Next, with reference to a flow chart of Drawing 127 , processing of coding service which codes a picture inputted from the client computer 27 
performed by the coding server 17 is explained. In this explanation, a case where a user of coding service pays a utilization charge is explained. 
[0748]In Step S1801, the client computer 27 outputs information which specifies a picture to the coding server 17. That is, information which is a 
picture concrete as information which specifies a picture which a user wants to code, or specifies picture ID which specifies a picture, and an object 
is outputted to the coding server 17. 

[0749]In Step 3181 1, the coding server 17 acquires a specified picture. That is, when picture ID which specifies the picture when a picture has been 
transmitted from the client computer 27 has been transmitted, a picture corresponding to the picture ID is read and acquired from on the network 

1. 

[0750]In Step S1812 and S1821, the accounting part 17a of the coding server 17 and the fee collection server 24 perform accounting. About 
accounting, in Drawing 118 and Drawing 120 , since it is the same processing as a case in separation service, the explanation is omitted. 
[0751 ]In Step S1813, the separation sections 2002 of the coding server 17 perform picture separation. Since picture separation is the same as 
processing of Step S1013 of a flow chart of Drawing 117 , the explanation is omitted. 

[0752]The coc ing server 17 outputs a foreground ingredient picture and a background component image required in processing of Step S1813 to the 
storage serve's 18, and makes them memorize in Step S1814 (accumulation). In Step S1831, the storage servers 18 memorize a foreground 
ingredient picture and a background component image which have been transmitted. 

[0753]In Step S1815, the coding server 17 adds a motion vector and position information to foreground ingredient image position data and 
background component position information which were generated by coding, and transmits to the client computer 27. 

[0754]In Step S1802, the client computer 27 receives and memorizes foreground ingredient image position data, background component image 
position infornation, a motion vector, and position information which have been transmitted from the coding server 1 7. 

[0755]When ssparating an inputted picture, coding and coding a picture similar to an already coded picture, only data used as the difference is given 
to numerals ( mage position data) of an already coded picture, and it may be made for the coding part 2001 to output it. For example, encoded 
information of the 1st picture that consists of the foreground ingredient picture 1, the foreground ingredient picture 2, and encoded information of 
the mixture mtio 1 when combining a picture as shown with Drawing 128 , When compounding encoded information of the foreground ingredient 
picture 1 , the foreground ingredient picture 3, and the 2nd picture that consists of encoded information of the mixture ratio 2, the foreground 
ingredient picture 1, Since it is contained in any information on a picture, when compounding, encoded information of the foreground ingredient 
picture 1 of one picture may be omitted, and only a part from which information on the foreground ingredient picture 1 is deleted can raise a 
compression ratio rather than a case where it compounds simply as a result. 

[0756]When accumulating the 1st picture and 2nd picture as shown in Drawing 128 as a result and the 1st picture is accumulated previously, the 
2nd picture should accumulate only the foreground ingredient picture 3 used as difference, and encoded information of the mixture ratio 2. For this 
reason, when encoded information of the same picture is accumulated over plurality, the more the number of pictures accumulated increases, the 
more a compression ratio will improve. 

[0757]lt may oe the information which a user specified about the mixture ratio, a motion vector, and position information which are coded by the 
coding server 17 as shown in Drawing 129 . A picture made to code reads a foreground ingredient picture and a background component image 
corresponding to picture ID specified by user from the storage servers 1 8, and it may be made to code them, as shown in Drawing 129 . In this case, 
the separatio n sections 2002 need to be formed in the coding server 17. 

[0758] Although picture ID has been used as information which specifies a picture in this specification, it replaces with this and may be made to use 



image position data. 

[0759]. Next, the synthetic server 19 performs with reference to a flow chart of Drawing 130 . Information and a motion vector which specify the 
pictures A and B inputted from the client computer 27, the mixture ratio, position information, and processing of service which combines the 
pictures A anc B specified from the amount of Japanese quince adjustments by moving are explained. 

[0760]informai ion as which the client computer 27 specifies the pictures A and B in Step S1901. a motion vector, the mixture ratio, position 
information, and a motion — a Japanese quince — information on the amount of adjustments is outputted to the synthetic server 19. Namely. 
[ whether it is a concrete picture as information which specifies the pictures A and B which a user wants to combine, and ] or picture A-1D and 
B-ID (coded nbove-mentioned image position data may be sufficient) which specify the pictures A and B, a motion vector, the mixture ratio, 
position information, and a motion — a Japanese quince — information on the amount of adjustments is outputted to the synthetic server 19. 
[0761]In Step S1911, the synthetic server 16 acquires a specified picture. That is. when picture ID (coded above-mentioned image position data 
may be sufficient) which specifies the picture when a picture has been transmitted from the client computer 27 has been transmitted, a picture 
corresponding to the picture ID is read and acquired from on the network 1. 

[0762]In Step S1912 and S1921, the accounting part 19a of the synthetic server 19 and the fee collection server 24 perform accounting. About 
accounting, in Drawing 118 and Drawing 120 . since it is the same processing as a case in separation service, the explanation is omitted. The 
above-mentioned coding server 16 is used for this accounting, and when a user who received coding service has paid, it can omit it. On the contrary, 
a user who received synthetic service may be made to pay instead of a user who received coding service. 

[0763]in Step S1913 — the synthetic server 19 — a motion vector, the mixture ratio, position information, and a motion — a Japanese quince — 
processing of composition of the pictures A and B is performed based on information on the amount of adjustments. 

[0764]In Step S1914, the synthetic server 19 gives ID to image composing (A+B) required in processing of Step S1913. and transmits to the client 
computer 27. 

[0765]In Step S1902, the client computer 27 receives and memorizes image composing (A+B) transmitted from the synthetic server 19. 
[0766]Synthe:ic server 19 self is made to memorize the client computer 27, or image composing (A+B) transmitted by the synthetic server 19 is 
made to output to the storage servers 18 via the network 1, and it can make it memorize according to a user's instructions (accumulation). 
[0767]by the way — although the synthetic server 20 is made as [ be / combining two or more pictures as mentioned above / possible ] — this time 
— a motion — a Japanese quince — using the amount of adjustments as a key — a motion of image composing -- a Japanese quince can be added 
and encrypted images can be generated. Drawing 131 shows composition of the motion dotage adjunct 2021 for codes which makes the synthetic 
server 20 generate encrypted images and which is provided for accumulating. 

[0768]The input process part 2031 of the motion dotage adjunct 2021 for codes, Picture selection information which chooses a picture (background 
component image) which is going to move information which serves as a cryptographic key at the imaging part 2032 in an inputted encrypted signal 
which it is going to encipher, and is going to compound the Japanese quince preparing part 2033 and an encrypted signal as a foreground ingredient 
picture is outoutted to the synthetic server 20, respectively. 

[0769]When sn encrypted signal inputted from the input process part 2031 is not a picture signal, the imaging part 2032 changes the signal into a 
picture signal moves, and is outputted to the Japanese quince adjunct 2033. That is, since it becomes conditions that it is a picture signal, in order 
to make a signal by which encryption processing is carried out correspond to the processing, it images a signal which is not a picture signal. 
[0770]The motion dotage preparing part 2033 moves based on information of speed and a direction which were inputted from the input process part 
2031, generates the amount of Japanese quince adjustments, moves to a picture signal inputted from the imaging part 2032, performs Japanese 
quince attacked processing, and outputs it to the synthetic server 20. The synthetic server 20 acquires a background component image based on 
picture selection information inputted from the input process part 2031, by making into a foreground ingredient picture a picture which moves and 
is further inputted from the Japanese quince adjunct 2033, compounds with an acquired background component image, and generates and displays 
image compcsing. At this time, the background component image itself may be sufficient as picture selection information which specifies a 



background component image, and it may be background component image position information or background component image ID. 
[0771]Next. with reference to Drawing 132 , image composing enciphered by the motion dotage adjunct 2021 for codes provided in the synthetic 
server 20 is docoded, and the motion dotage removing part 2041 for codes changed into a signal of a dimension is explained, a motion for codes 
shown in Draw ng 131 and Drawing 132 — a Japanese quince — the adjunct 2021 and a motion for codes — a Japanese quince — it may carry out 
and the removing part 2041 may think [ which may be considered as a functional block diagram of software built in the client computer 27, for 
example ] tha': it is a block diagram of hardware, a motion for codes — a Japanese quince — the adjunct 2021 and a motion for codes — a 
Japanese quince — it may be made to constitute the removing part 2041 as a dedicated server on the network 1 

[0772]The separation server 1 1 outputs a foreground ingredient picture to which enciphered image composing was divided into a foreground 
ingredient picture and a background component image, it moved and a Japanese quince was added to the input process part 2051. 
[0773]lf information on speed and a direction is inputted as a key for decoding a foreground ingredient image which was inputted from the 
separation server 1 1 and which is enciphered, and its enciphered foreground ingredient image, the input process part 2051 will move and will be 
outputted to the Japanese quince removing part 2052. Speed and a direction used as this key are set as each, when two dimensions of a x direction 
and a y direction show a picture. 

[0774]Based on information on speed inputted from the input process part 2051, and a direction, the motion dotage removing part 2052, a 
foreground ingredient image which generated motion blur quantity and was enciphered — a motion for codes — a Japanese quince — a motion 
given by the adjunct 2021 — a Japanese quince — a motion contrary to attached processing — a Japanese quince — attached processing is 
performed, an enciphered foreground ingredient image is decoded, and it outputs to the signal conversion section 2053. When an encrypted signal is 
not a picture, the signal conversion section 2053 changes into the original signal a picture signal which moved and was inputted from the Japanese 
quince removing part 2052, and outputs it. 

[0775]namely, the above-mentioned motion — a Japanese quince — the preparing part 2033 ( Drawing 131 ) and a motion — a Japanese quince — 
the removing part 2052 — substantial — a motion of drawing 97 — a Japanese quince -- performing the same processing as the adjunct 806, 
moving as a cryptographic key, and using the amount of Japanese quince adjustments — a motion reverse to mutual — attached processing of a 
Japanese quince is performed. However, points which a x direction mentioned later or a y direction moved, and were previously performed among 
Japanese quir ce attached processing of moving and performing processing of a gain rise to Japanese quince attached processing differ. 
[0776]By moving here and adding a Japanese quince explains a principle which enciphers a picture signal. For example, as shown in Drawing 133, 
when a photographic subject moves in the direction of an arrow, If this is picturized by the sensor 76a which consists of CCD etc., before and after 
the move direction, a picturized picture will move and a mixing zone (a covered background region and an uncovered background region) will produce 
as a Japanest quince (for details, see drawing 31 ). According to that speed, it moves, a field of a Japanese quince spreads, and Drawing 134 shows 
signs that a zo\or picturized is spread further, while a photographic subject is moving it rightward [ left-in-the-figure ]. when picturizing a 
photographic subject as shown an example which shows this phenomenon and shown with Drawing 134 (A) by the sensor 76a. That is, while a 
photographic subject is moving to a longitudinal direction by speed v, a picture as shown with Drawing 134 (B) shall be picturized. Supposing it is the 
field aO thru/or a0", a field picturized at this time while a photographic subject is standing it still, a field where a photographic subject of Drawing 134 
(B) is picturized becomes the field a1 thru/or a1 ', and the original position aO thru/or a color of a field of a0' become thin — a motion — a color has 
spread in the Fields a1 thru/or aO used as a Japanese quince and field aO' thru/or a1\ the same — carrying out — a photographic subject — speed 

— two — v O.wice of speed v) — moving — **** — if ~ a figure — 1 34 — (~ C — ) — being shown — as — further — moving — a Japanese 
quince — becoming — a field — a — two — or — a — zero — and — a field — a — zero — ' — or — a — two — ' — a color — spreading — 
things — being shown — having — **** . If a photographic subject moves with the speed 3v. as shown in Drawing 134 (D), a motion — when it 
spreads in th<: fields a3 thru/or aO used as a Japanese quince and field aO' thru/or a3' and moves with the speed 4v, Drawing 1 34 (E) shows ~ as 

— a motion — it spreads in the fields a4 thru/or aO used as a Japanese quince and field aO' thru/or a4', and a color becomes thin as a whole. 
Namely, each pixel value outputted from the sensor 76a, Since it is the result of integrating with a certain portion into which an object to picturize 



has breadth spatially about shutter time, whenever speed of a pixel value with which it integrates generally of a photographic subject increases only 
a part not cha iging, a color will be thinly picturized by spreading spatially. Therefore, decipherment of a photographic subject becomes difficult as a 
color becomei, thin, a field spreads, it moves and a field of a Japanese quince becomes large. Here, a color shows the possibility of decipherment, 
and it shows that the possibility of decipherment becomes small, so that its possibility of decipherment is so large that it is deep and it is thin. 
[0777]a motion — a Japanese quince — encryption by adjustment uses this character and cannot take place in the real world — a motion of the 
direction of two dimensions — a Japanese quince is made to encipher by producing a picture That is, as Drawing 135 shows, a picture by which a 
black dot-like photographic subject was picturized is shown in the leftmost sequence highest rung of a matrix in the state where there is no motion, 
a motion of tr e state [ state / this ] where a motion to a lengthwise direction is for example, — if a Japanese quince is added, a black dot-like 
photographic iubject will serve as a picture which was moved to a sliding direction and a Japanese quince produced as shown in the middle line 
highest rung. IF it moves to a transverse direction and a Japanese quince is produced, as shown in the middle line middle, it will become the picture 
which was moved in the direction of four directions of a photographic subject, and a Japanese quince produced. 

[0778]It is in ;his state, and if a motion (speed) of a longitudinal direction is enlarged, it moves further again and a Japanese quince is added, as 
shown in the middle line bottom, it will become the picture in which it moved to a longitudinal direction and a Japanese quince field spread further. 
If it moves to a sliding direction and this picture is made to produce a Japanese quince further, as shown in the rightmost sequence bottom, a black 
dot-like photographic subject will move, a Japanese quince field will spread, and a color will become thin as a whole. Since the possibility of 
decipherment of a photographic subject falls as a result it becomes possible to make the picture itself encipher. 

[0779]Next, with reference to a flow chart of Drawing 136 , encryption processing which the motion dotage adjunct 2021 for codes moved, and used 
the amount of Japanese quince adjustments is explained. The following explanation explains an example which enciphers a picture by which a 
photographic subject which consists of 5x5 pixels was picturized, as shown in Drawing 137 . Here, in Drawing 137 , it supposes that it is shown with 
the pixel values a thru/or y of each 5x5-pixel pixel, and y shows a lengthwise direction, x shows a transverse direction, and a time axis is explained 
as the time t. 

[0780]In Step S2001, when the input process part 2031 judged whether an encrypted signal was inputted, and the processing is repeated and it is 
judged with having been inputted until it is inputted, the processing follows it to Step S2002. 

[0781]In Step S2002, the input process part 2031 outputs an inputted encrypted signal to the imaging part 2032. In Step S2003, the imaging part 
2032 judges whether an inputted encrypted signal is a picture signal. For example, when judged with an encrypted signal not being a picture signal, 
in Step S2004, the imaging part 2032 changes an encrypted signal into a picture signal, moves, and is outputted to the Japanese quince adjunct 
2033. In Step S2003, when judged with encrypted information being a picture signal, the imaging part 2032 moves as it is, and outputs an inputted 
encrypted signal to the Japanese quince adjunct 2033. 

[0782]In Step S2005, if it judges whether information on speed used as a key and a direction was inputted, and the processing is repeated and a key 
of speed and a direction is inputted until it is inputted, the processing will follow the input signal treating part 2031 to Step S2006. 
[0783]In Step S2006, it moves and the Japanese quince adjunct 2033 is enciphered about a x direction to an inputted picture signal (motion dotage 
is added). 

[0784]Here, a generation method of a concrete pixel value when moving a photographic subject and enciphering by Japanese quince adjustment is 
explained with reference to Drawing 137 thru/or Drawing 149 . 

[0785]Here, as shown in Drawing 137 , how to move the pixels a thru/or e of the bottom to a x direction, make produce a Japanese quince, and 
encipher is e> plained. When movement quantity v which shows speed used as a key at this time is set to 5 (it is the virtual number of partitions 5), 
a pixel of the bottom shown with Drawing 138 is shown like Drawing 139 . Namely, since a pixel value of each time direction is divided into five, A 
relation of a/5=a0=a1=a2=a3=a4, b/5=b0=b1=b2=b3=b4, c/5=c0=c1=c2=c3=c4, d/5=d0=d1=d2=d3=d4, and e/5=e0=e1=e2=e3=e4 will be filled. Here, 
it is a pixel v«,lue of the last time as a pixel in a figure 139 Nakagami stage. 

[0786]If a motion (in the case of now the direction of figure Nakamigi) of a x direction is given to a photographic subject, arrangement of a pixel 



value will be si d with a predetermined time interval, and will turn into arrangement as shown with Drawing 140 as a result. Namely, in timing [ begin ] 
which moves, the pixel values aO thru/or eO are the original positions, and are the following timing, The pixel values a1 thru/or e1 are slid rightward 
by 1 pixel, and are the following timing further, The pixel values a2 thru/or e2 are slid rightward by 1 more pixel, and are the following timing further. 
The pixel values a3 thru/or e3 are slid rightward by 1 more pixel, are the following timing further and serve as arrangement which a pixel value 
moved according to a motion of a photographic subject as it said that the pixel values a4 thru/or e4 were slid rightward by 1 more pixel. 
[0787]Each pixel value on a xy plane will be referred to as having added a pixel value shown with Drawing 140 to a time direction. However, for 
example, wher a leftmost sequence or a rightmost sequence has a possibility of it being set only to the pixel value aO and e4, and becoming what 
has a very sm;ill value as a pixel value and same processing is carried out also to a y direction after this, a gain rise is processed so that it may not 
become a very small value. An example which processed this gain rise is shown in Drawing 141 . 

[0788]Here. a0*=5xa0, b0*=. (5/2) xbO, a0*=. (5/2) xa1. c0*= (5/3) xcO, b1*= (5/3) xb1, a2*=. (5/3) xa2. d0*=. (5/4) xdO. d*=. (5/4) xd, b2*=. 
(5/4) xb2. a3*=. (5/4) xa3, e1*=. (5/4) xe1, d2*=(5/4) xd2, c3*=(5/4) xc3, b4*=(5/4) xb4, e2*=(5/3) xe2, d3*=(5/3) xd3, c4*=(5/3) xc4, e3*=(5/2) 
xe3, and d4*=<5/2) xd4 — and, It is e4*=5xe4. That is, an aggregate value of each pixel is adjusting a gain so that it may become a pixel value for 1 
pixel. As a result, the pixels a thru/or e shown with Drawing 138 will be changed into the pixel ax thru/or dx' as shown with Drawing 1 42 if movement 
quantity v is enciphered by x direction on condition of 5 (enciphered), and (if motion dotage addition is carried out) a horizontal pixel number of a 
photographic subject will increase to 9 from 5. Here a pixel value ax=ax*. bx=(b0*)+ (a1*). cx=(c0*) +(b1*)+ (a2*). dx=(d0*) +(c1*) +(b2*)+ (a3*), 
They are ex=( 2 0) +(d1) +(c2) +(b3)+ (a4), ax'=(e1*) +(d2*) +(c3*)+ (b4*), bx'=(e2*) +(d3*)+ (c4*). cx'=(e3*)+ (d4*). and ex=ex*. 
[0789]When it enciphers to a x direction to 5x5 pixels [ which showed the above processings with Drawing 137 ] all the y, a pixel value as shown with 
Drawing 143 will be calculated, namely, — a pixel — ax — or — yx — and — a pixel — ax — ' — or — dx — " fx — ' — or ix — ' — kx — 
' — or — nx - - ' — px — ' — or — sx — ' — and — ux -- ' — or -- xx — ' — a pixel — asking — having — ax direction -- moving — a Japanese 
quince — procJucing — things — ax direction — breadth — producing — nine — a pixel — every — asking — having — ******** 
[0790]Here, it returns to explanation of a flow chart of Drawing 136 . 

[0791 ]In Step S2007, it moves and the Japanese quince adjunct 2033 enciphers a picture signal coded to a x direction to a y direction. 
[0792]Here, as shown in Drawing 144 , how to move the pixel ax of a rightmost sequence shown with Drawing 143 , fx, kx, px, and ux to a y direction, 
make produce a Japanese quince, and encipher is explained. When movement quantity v which shows speed used as a key at this time is set to 5 (it 
is the virtual r umber of partitions 5), a pixel of a rightmost sequence shown with Drawing 143 is shown like Drawing 144 . Namely, since a pixel value 
of each time direction is divided into five. ax/5=ax0=ax1=ax2=ax3=ax4, A relation of fx/5=fx0=fx1=fx2=fx3=fx4, kx/5=kx0=kx1=kx2=kx3=kx4. 
px/5=px0=px1=px2=px3=px4, and ux/5=ux0=ux1=ux2=ux3=ux4 will be filled. Here, it is a pixel value of the last time as a pixel in a figure 145 
Nakagami staj;e. 

[0793]If a motion is given to a y direction to a photographic subject, arrangement of a pixel value will be slid with a predetermined time interval, and 
will turn into arrangement as shown with Drawing 146 as a result. Namely, in timing [ begin ] which moves, pixel value axO, fxO, kxO, pxO, and uxO are 
the original positions, and they are the following timing, Pixel value ax1, fx1 , kx1. px1, and ux1 are slid rightward by 1 pixel, and they are the following 
timing further, Pixel value ax2, fx2, kx2, px2, and ux2 are slid by 1 more pixel, and they are the following timing further rightward, Pixel value ax3, fx3, 
kx3, px3, and ux3 are slid by 1 more pixel, and they are the following timing further rightward, Pixel value ax4, fx4, kx4, px4, and ux4 become the 
arrangement which a pixel value moved according to a motion of a photographic subject as it said that it slid by 1 more pixel rightward. 
[0794]Here, it returns to explanation of a flow chart of Drawing 136 . 

[0795]In Step S2008, the synthetic server 19 compounds a background component image compounded with an enciphered picture (foreground 
ingredient picture). For example, if a background component image (picture which becomes a x direction from a pixel for one step) which consists of 
a pixel value cf the pixel values BO thru/or B9 arranged in a y direction as shown with Drawing 147 is compounded, a value to which a pixel value as 
shown with D awing 148 was added will turn into a pixel value. That is, a pixel value of a picture by which a pixel value of a picture (motion dotage 
was added) enciphered in the xy direction is compounded as a pixel value of a foreground ingredient picture is compounded as a pixel value of a 



background component image, a result — ****** — a figure — 149 — being shown — as — a pixel value — A — F — K — P — U — Ay 

Fy - — K/ __ • __ p y — ' — asking — having — ******** — each — a pixel value. Pixel value A=ax0+B0x4/5, and pixel value 

F=fx0+ax0+B1 <3/5, and pixel value K=kx0+fx1+ax2+B-2x2/5, and pixel value P=px0+kx1+fx2+ax3+B3x1/5, and pixel value U=ux0+px1+kx2+fx3+ax4, 

It is set to pixel value Ay'=B5x1/5+ux1+px2+kx3+fx4, pixel value Fy'=B6x2/5+ux2+px3+kx4, pixel value Ky '=B7x3/5+ux3+px4, and pixel value Py' 

=B8x4/5+ux4. 

[0796]By performing these processings to all the y directions, image composing which uses an enciphered foreground ingredient picture as shown 
with Drawing 50 as a background component image is generated, namely, the inputted 5x5-pixel picture — 9x9 pixels (the pixels A thru/or Y.) a 
pixel — Ax — or — Dx — a pixel — Fx — or ~ Ix -- a pixel — Kx — or — Nx -- a pixel — Px — or — Sx — a pixel ~ Ux — or — Xx — a pixel 
„ Ay __ - or — T y - — a pi xe | — Ax — ' — or — Dx — ' — a pixel — Fx — ' — or — Ix — ' — a pixel — Kx — * — or — Nx — ' — and 
— a pixel — Px — ' — or — Sx — ' — a picture — changing — having . 

[0797]in addition — processing of decoding — a motion for codes — a Japanese quince — in the removing part 2041 -- a motion for codes — a 
Japanese quir ce — a motion completely contrary to processing of the adjunct 2021 — a Japanese quince — since attached processing will be 
performed, the explanation is omitted. 

[0798]ln processing of the above-mentioned step S2006, when enciphering to a x direction, after performing gain rise processing, encryption of a y 
direction is performed. Therefore, after decoding to a y direction, since decoding brings down a gain, it is necessary to decode it to a x direction. 
Since processing of a gain rise is made to a direction enciphered previously, it is necessary to make it correspond to an order that decoding was 
also enciphered, although an order of encryption processing of a y direction and a x direction may be replaced. 

[0799]Next, with reference to a flow chart of Drawing 151 , encryption service by the synthetic server 19 equipped with the motion dotage adjunct 
2021 for encryption which is Drawing 131 and was shown is explained. This processing is processing in a case of the client computer 27-1 
connected on the network 1 transmitting an encrypted signal to the synthetic server 19, and making this encipher and making it transmit to the 
client computer 27-2. moreover — the client computer 27 — a motion for codes — a Japanese quince — hardware provided with a separation 
function of a aicture of the separation server 1 1 which has the removing part 2041 shall be provided, or software shall be installed 
[0800]In Step S2101, the client computer 27-1 transmits information (encrypted signal) which it is going to encipher, information on speed used as 
a cryptographic key, and a direction, and picture selection information (information which chooses a background component image) to the synthetic 
server 19. 

[0801]In Step S21 11, synthetic Saba's 19 motion dotage adjunct 2021 for codes enciphers information (encrypted signal) which was inputted from 
the client conputer 27-1 and which it is going to encipher based on a cryptographic key. and performs encryption processing which compounds a 
selected bacl' ground component image. About encryption processing, since it explained with reference to a flow chart of Drawing 136, it omits. 
[0802]ln Step S2112, the synthetic server 19 transmits a picture enciphered and combined to the client computer 27-1 by moving and adding a 
Japanese qui ice. 

[0803]ln Step S2102, the client computer 27-1, When image composing received from the synthetic server 19 is displayed, a user judges whether it 
is a picture considered as a request and it is judged with it being a desired picture, in Step S2103, it is notified to the synthetic server 19 that it was 
a desired picture. In Step S21 13, the synthetic server 19 judges whether it was a desired picture, for example, since it has received notice that it is 
a desired pict ure, in Step S2103 in now, the processing follows it to Step S2114. 

[0804]In Step S2114, the accounting part 19a of the synthetic server 19 performs accounting with the fee collection server 24. About accounting. 
in Drawing 1 1 8 and Drawing 120 , since it is the same processing as a case in separation service, the explanation is omitted. 

[0805]In Step S2115, the synthetic server 19 transmits enciphered image composing to the client computer 27-1. In Step S2104, the client 
computer 27 -1 receives enciphered image composing, and transmits to the client computer 27-2 as it is. 

[0806]In Step S2141, the client computer 27-2 receives enciphered image composing. In Step S2142, the client computer 27-2 judges whether a 
key was inpu ted or not, and it repeats the processing until a cryptographic key is inputted. In Step S2142, as a cryptographic key, if information on 



speed and a direction is inputted, based on speed and a direction which were inputted, it will move, the Japanese quince solvent wiping removal part 
2041 will movt , and Japanese quince processing will be performed in Step S2143. In Step S2144, a picture from which it moved and a Japanese 
quince was removed is displayed. 

[0807]ln Step 52102, when judged with it not being a desired picture, in Step S2015, it notifies the synthetic server 19 that it is not a desired picture, 
and the processing returns to Step S2101. By this processing, in Step S21 14, since it will be judged with it not being a desired picture, processing 
of the synthetic server 1 9 returns to processing of Step S21 11 . 

[0808]That is, when a key of speed as which a user of the client computer 27-2 was specified by user of the client computer 27-1, and a direction 
is correctly inputted by this processing, an image by which an enciphered image was decoded correctly is displayed. Decoding service can also be 
provided by the same system as above-mentioned encryption service. 
[0809]Next, tre correction server 20 is explained with reference to Drawing 152 . 

[0810]The separation sections 11 of the correction server 20 divide an inputted picture (picture which read a picture corresponding from the 
network 1 top in the case of a picture which picture ID may be sufficient as and was specified by picture ID) into a foreground ingredient picture and 
a background component image, A foreground ingredient picture is moved and a background component image is outputted to the Japanese quince 
controller 2101 at a synchronizer. The motion dotage controller 2101 moves by motion (grade of correction is adjusted) blur quantity which had an 
inputted foreground ingredient picture specified, fades, is adjusted, and is outputted to the synchronizer 2101. The synchronizer 2101 compounds a 
foreground ingredient picture by which motion dotage adjustment was carried out, and an inputted background component image, and outputs it as 
a corrected image. 

[0811]For example, a picture as shown with Drawing 153 (A) shall be inputted into the correction server 20. That is. as shown in Drawing 153 (A) 
right part, whon a foreground moves a background top to an arrow direction, it moves to a direction of movement and an opposite direction of a 
foreground, and a Japanese quince arises. As a portion of this motion dotage is a mixing zone and it is shown in Drawing 153 (A) left part, A mixing 
zone produced in a front portion of the move direction is CB (Covered Background), and a mixing zone produced in a rear part of the move direction 
is UB (Uncovered Background). With a figure of Drawing 153 (A) left part, since the time-axis t is set as a lengthwise direction, storage states of a 
pixel value on a pixel and a relation of the passage of time are shown with movement. The separation sections 1 1 divide this inputted image into a 
foreground and a background, as shown in Drawing 153 (B). At this time, a mixing zone of an inputted picture is also extracted simultaneously. 
[081 2]A foreground ingredient picture as shown with Drawing 153 (B) moves, and the motion dotage controller 2101 adjusts a Japanese quince, for 
example, generates a foreground ingredient picture as shown in Drawing 153 (C). That is, in now, it moves and a Japanese quince is made small 
(portions of GB and UB are made small), a motion — a motion for adjusting a Japanese quince — a Japanese quince — while a user repeats 
operation several times, it may be made to input the amount of adjustments — carrying out — a motion — a Japanese quince it may be made 
for the controller 2101 to adjust to a predetermined value 

[081 3]a moticn as the synchronizer 2102 compounded a foreground ingredient picture adjusted as Drawing 153 showed, and an inputted background 
component iriage and shown with Drawing 153 (D) — a Japanese quince — an adjusted foreground ingredient picture is combined with a 
background component image, and is outputted. 

[0814]A sepa-ated background component image is not outputted to the synchronizer 2102 but a background component image to make it changing 
is inputted irto the synchronizer 2102 to change into other different background component images from a thing of a picture into which a 
background component image was inputted, the correction server 20 — the separation sections 1 1 and a motion — a Japanese quince — the 
controller 2101 and the synchronizer 2102 — the separation server 1 1 on the network 1 , and a motion — a Japanese quince — it transposes to the 
adjustment server 16 and the synthetic server 19. and may be made to constitute 

[081 5]Next, \a ith reference to a flow chart of Drawing 154 , processing of correction service performed by the correction server 20 in which a picture 
inputted from the client computer 27 is corrected is explained. 

[0816]In Step S2201, the client computer 27 outputs information which specifies a picture to the correction server 20. That is, picture ID which is 



a concrete picture or specifies a picture as information which specifies a picture which a user wants to correct is outputted to the correction 
server 20. 

[081 7]In Step 3221 1, the correction server 20 acquires a picture and a background component image which were specified and which it is going to 
correct, and tre separation sections 1 1 divide into a foreground ingredient picture and a background component image a picture which it is going to 
correct. That i 5, when picture ID which specifies the picture when a picture has been transmitted from the client computer 27 has been transmitted, 
a picture corre sponding to the picture ID is read and acquired from on the network 1. The separation sections 1 1 divide an acquired picture into a 
foreground ingredient picture and a background component image. 

[0818]In Step S2212 and S2221, the accounting part 20a of the correction server 20 and the fee collection server 24 perform accounting. About 
accounting, in Drawing 118 and Drawing 120 . since it is the same processing as a case in separation service, the explanation is omitted. 
[0819]In Step S2213, the correction server 20 moves, a foreground ingredient picture moves and the Japanese quince controller 2101 performs 
regulated treatment of a Japanese quince. Since regulated treatment of motion dotage is the same as processing explained with reference to a flow 
chart of Drawing 104 . the explanation is omitted. 

[0820]In Step S2214, the synchronizer 2102 compounds a background component image specified as a foreground ingredient picture by which 
motion dotage adjustment was carried out. In Step S2215, the correction server 20 transmits image composing required in processing of Step 
S2214, i.e., a corrected image, to the client computer 27. 

[0821]In Step S2202, the client computer 27 receives and memorizes a corrected image transmitted from the correction server 20. 
[0822]Correction server 20 self is made to memorize the client computer 27, or a picture corrected by the correction server 20 is made to output 
to the storage servers 18 via a network, and it can make it memorize according to a user's instructions (accumulation). 

[0823]Next, wth reference to a flow chart of Drawing 155 , processing of purchase service of a picture which purchases a picture specified from the 
client computer 27 performed by the purchase server 21 is explained. 

[0824]In Step S2301, the client computer 27 outputs information which specifies a picture which wishes to purchase to the purchase server 21. 
That is, picture ID which specifies a picture as information which specifies a picture which a user wants to purchase is outputted to the purchase 
server 21. 

[0825]In Step S2311, the purchase server 21 acquires a picture which wishes to purchase. That is, a picture corresponding to picture ID 
transmitted from the client computer 27 is read and acquired from on the network 1. 

[0826]In Step S2312 and S2321, the accounting part 21a of the purchase server 21 and the fee collection server 24 perform accounting. About 
accounting, in Drawing 118 and Drawing 1 20 , since it is the same processing as a case in separation service, the explanation is omitted. 
[0827]In Step S2313, the purchase server 21 transmits a picture acquired by processing of Step S2311 to the client computer 27. 
[0828]In Step S2302, the client computer 27 receives and memorizes a picture transmitted from the purchase server 21. 

[0829]Purcha;e server 21 self is made to memorize the client computer 27, or a picture purchased by the purchase server 21 is made to output to 
the storage servers 18, and it can make it memorize according to a user's instructions (accumulation). What gives a picture can be performed by 
enabling it to transmit to other client computers 27, for example. Other users become possible [ purchasing a foreground ingredient picture 
separated, respectively, a background component image, compounded image composing, or a corrected corrected image by separation service, 
synthetic service, or correction service as mentioned above ]. 

[0830]Next, with reference to a flow chart of Drawing 156 , processing of sale service of a picture performed by the sale server 22 in which a picture 
specified from the client computer 27 is sold off is explained. 

[0831]In Step S2401 , the client computer 27 outputs information which specifies a picture which wishes to sell off to the purchase server 21. That 
is. a picture which a user wants to sell off is outputted to the sale server 22. 

[0832]In Step S2411, the sale server 22 acquires a picture which wishes to sell off. That is, a picture transmitted from the client computer 27 is 
acquired. 



[0833]In Step S2422, the sale server 22 sets up a suitable price to a picture expected of sale. Setting out of a price may be beforehand set up by 
user, may mahe it set it up in auction form on the network 1, for example, and further, When it is the picture by which a person was picturized, a 
method which a person currently picturized sets up by whether you are predetermined celebrities, for example may be used. 
[0834]ln Step S2413 and S2431, the accounting part 22a of the sale server 22 and the fee collection server 24 perform sale accounting. 
[0835]Here, su\e accounting is explained with reference to a flow chart of Drawing 157 . Although actual sale accounting is performed by the sale 
server 22 and the fee collection server 24, since it is outputted also from the client computer 27, also about processing of the client computer 27, 
it doubles and information required for various kinds of processings is explained here. 

[0836]In Step S2501, ID information which identifies a user (user who requests sale of a picture) is transmitted to the sale server 22 via the network 
1. 

[0837]In Step S251 1, the sale server 22 transmits the amount of money for sale, and ID which identifies the sale server 22 to the fee collection 
server 24 bas«id on ID information transmitted from the client computer 27. 

[0838]In Step S2521. the fee collection server 24 requests transfer to the financial server 25 of a customer account corresponding to ID 
information fo- purchased amount to the financial server 26 with a donor account based on ID which identifies the transmitted sale server 22. 
[0839]In Step S2531, the financial server 26 for donors transfers the amount of money equivalent to the amount of money for sale to the financial 
server 25 for customers by which a customer's account is established from a donor's account. 
[0840]Here. it returns to explanation of a flow chart of Drawing 156 . 

[0841]In Step S2424, the sale server 22 notifies the client computer 27 that sale was completed. In Step S2402, the client computer 27 receives a 
notice of the completion of sale. 

[0842]May memorize a picture sold off by user to sale server 21 self, or it is made to output to the storage servers 18, and the sale server 22 can 
make it memorize (accumulation). When setting up a price in auction form as mentioned above, it may be made to transmit to a successful 
tenderer's cli< nt computer 27. 

[0843]Next, the retrieval server 23 is explained with reference to Drawing 158 . 

[0844]The retrieval server 23 searches a picture currently picturized on the camera terminal device 1 connected on the network 1 based on a 
search condit on (image request signal) inputted from the client computer 27 etc., and outputs a demand picture. As a search condition, they are 
time, a season, the weather, an area, a place, or a photographic subject. 

[0845]The control section 2161 of the retrieval server 23 controls operation of the whole retrieval server 23. It corresponds to camera ID for every 
camera terminal device 28 which retrieval server 23 self connected to the network 1 recognizes in the database 2162. The data 2162b of position 
data (acquired by GPS76b which it had camera terminal device 28) of each camera terminal device 28, a meteorological data, data of a photographic 
subject curreitly picturized, etc. is memorized as a database. The contents of this database 2162 are predetermined time intervals, the control 
section 2161 controls the communications department 2165, acquires them from each camera terminal device 28 via the network 1. and they are 
updated. 

[0846]The storage parts store 2163 memorizes a picture acquired from the camera terminal device 28 on the network 1 from the communications 
department 2165, or memorizes information required for processing of various kinds of pictures. 

[0847]The demand information generating part 2164 arranges a search condition inputted from the client computer 27 on the network 1, etc., and 
generates conditions which can actually be searched on the database 2162. That is, for example, when a season is inputted as a search condition, 
a season can be specified by time information calculated by position data and the time operation part 2166 of each camera terminal device 28. Then, 
the demand information generating part 2164 generates position data of latitude longitude on the earth in which a season serves as "spring" from 
the time pressnt now, for example, when a search condition of "spring" is inputted. The control section 2161 acquires a picture [ / in "spring" ] by 
controlling ths communications department 2165 and reading an image pick of the camera terminal device 28 of camera ID corresponding to this 
position data from on the network 1 . 



[0848]The separation sections 2167 acquire a retrieval object's picture included in a read picture by separation. The separation sections 2167 have 
the same funcJon as the separation sections 11. 

[0849]Next, with reference to a flow chart of Drawing 159 , processing of a search service which searches a picture based on a search condition 
inputted from he client computer 27 performed by the retrieval server 23 is explained. 

[0850]In Step S2601. the client computer 27 outputs a search condition to the retrieval server 23. In Step S261 1, the retrieval server 23 receives 
a search cond tion by the communications department 2165. 

[0851]ln Step S2612 and S2631, the accounting part 23a and the fee collection server 24 of the retrieval server 23 perform accounting. About 
accounting, in Drawing 1 18 and Drawing 120 , since it is the same processing as a case in separation service, the explanation is omitted. Step S2612 
and accounting in processing of S2631 are the accounting about a fee to performing a search. 

[0852]ln Step S2613, the retrieval server 23 searches a picture corresponding to a search condition, and calls a picture which corresponds a picture 
corresponding in Step S2614. In Step S2641, the camera terminal device 28 transmits a picture picturized now to the retrieval server 23. 
[0853]That is, for example, as shown in Drawing 160 , it is assumed that the client computer 27-1 thru/or 27-5, the retrieval server 23 and the 
camera termiral device 28-1 thru/or 28-5 were connected to the network 1. At this time, the client computer 27-2 is operated by user, and as a 
search condit on, If "human being", a "car", and a "building" are transmitted by processing of Step S2611, the retrieval server 23 will search 
"human being ', a "car", and a photographic subject of a "building" with the database 2162 as a search condition in Step S2613. Namely, in the case 
of Drawing 160 , the retrieval server 23, the car 2172 -- the camera terminal device 28-1 of camera ID=1, and the human being 2183 — the camera 
terminal device 28-2 of camera ID=2 — and. It searches that the building 221 1 is picturized with the camera terminal device 28-5 of camera ID=5, 
and a picture is acquired from each camera terminal device 28 in Step S2614. 

[0854]ln Step S2515, a called picture is separated or the retrieval server 23 judges whether pictures (object) other than a desired condition are 
included. 

[0855]In the case of Drawing 1 60 , in a picture transmitted from the camera terminal device 28-1. Since clouds 2172 other than a search condition 
are contained houses 2181 other than a search condition are included in a picture transmitted from the camera terminal device 28-2 and a picture 
corresponding to a search condition in these pictures will be separated, the processing progresses to Step S2616. 

[0856]In Step S2616, the separation sections 2167 perform separation. Since separation is the same as processing of Step S1013 of a flow chart of 
Drawing 117 , the explanation is omitted. 

[0857]In Step S2617, a called picture is combined and it transmits to the client computer 27. In Step S2602, the client computer 27 acquires a 
picture transmitted from the retrieval server 23. In Step S2603, the client computer 27 judges whether a received picture was a desired picture. As 
shown in Drawing 160 , in the case of a picture displayed on display 27a-1, "human being" as a search condition, a "car", and a "building", Since it 
is contained in a picture as the human being 2182, the car 2172, and the building 2211 and is a desired picture, in Step S2604, it is notified to a 
retrieval servar that it was a desired picture. 

[0858]In Step S2618, the retrieval server 23 judges whether a notice transmitted from the client computer 27 was a desired picture. Since it was a 
desired pictur e in the case of Drawing 160 , the processing progresses to Step S2619. 

[0859]In Step S2619 and S2632. the accounting part 23a and the fee collection server 24 of the retrieval server 23 perform accounting. About 
accounting, ir Drawing 1 18 and Drawing 120 , since it is the same processing as a case in separation service, the explanation is omitted. Step S2619 
and accounting in processing of S2632 are the accounting about a fee to having transmitted a searched picture. In Step S2515, when it is only a 
picture of a search condition altogether, processing of Step S2616 will be skipped. 

[0860]In Step S2603, in spite of having specified a "house", "clouds", and a "face" as a search condition from the client computer 27-4, Since it 
will not be a desired picture when only the house 2181 and the clouds 2071 are displayed and the face 2201 is not displayed like a picture displayed 
on display 2 'a-4, The processing progresses to Step S2605, it transmits that it is not a desired picture to the retrieval server 23, and the 
processing is ended. 



[0861]Since it is reported at this time that the retrieval server 23 was not a desired picture in Step S2618, that processing is ended. 
[0862]In this case, it will not be paid to having transmitted a searched picture although a fee concerning retrieval processing will be paid. 
[0863]it becomes possible to make it encipher by according to the above, moving to an image and adding a Japanese quince — further a motion 
— removal of 3 Japanese quince, i.e., a motion added at the time of encryption, — a motion contrary to a Japanese quince — it becomes possible 
to decode an enciphered image by adding a Japanese quince. 

[0864]A recording medium which recorded a program which performs signal processing of this invention, . As shown in drawing 2 and 3. apart from 
a computer, are distributed in order to provide a user with a program. The magnetic disks 61 and 91 (a flexible disk is included) with which a program 
is recorded, the optical discs 62 and 92 (CD-ROM (Compact Disc-Read Only Memory).) . DVD (Digital Versatile Disc) is included. The 
magneto-optk al discs 63 and 93 (MD (Mini-Disc) (trademark) is included). Or it comprises ROMs 42 and 72 with which it is not only constituted by 
package medi*. which consist of the semiconductor memory 64 and 94 etc., but a user is provided in the state where it was beforehand included in 
a computer ard on which a program is recorded, a hard disk contained in the storage parts stores 48 and 78, etc. 

[0865]ln this specification, even if processing serially performed in accordance with an order that a step which describes a program recorded on a 
recording medium was indicated is not of course necessarily processed serially, it also includes a parallel target or processing performed individually. 
[0866]In this i pecification, a system expresses the whole device constituted by two or more devices. 
[0867] 

[Effect of the Invention]According to the 1st communication apparatus of this invention, a method, and the program, a user's demand information is 
inputted, The foreground ingredient picture which consists of a foreground object ingredient which constitutes the foreground object of the image 
data which consists of a pixel value determined according to the quantity of the light which constitutes the picture with which it integrated in time 
for every pixel according to demand information. The background component image which consists of a background object component which 
constitutes the background objects of image data is compounded, and it was made to output the image composing which generated and generated 
image composing. 

[0868]The co nmunications system and method of this invention input a user's demand information with the 1st communication apparatus. The 
image composing which transmits the inputted demand information to the 2nd communication apparatus, and is transmitted from the 2nd 
communication apparatus according to demand information is received, The 2nd communication apparatus receives the demand information 
transmitted from the 1st communication apparatus, The foreground ingredient picture which consists of a foreground object ingredient which 
constitutes the foreground object of the image data which consists of a pixel value determined according to the quantity of the light which 
constitutes the picture with which it integrated in time for every pixel according to demand information. The background component image which 
consists of a background object component which constitutes the background objects of image data was compounded, image composing is 
generated and the generated image composing was transmitted to the 1st communication apparatus. 

[0869]In the i!nd communication apparatus of this invention, the method, and the program, the demand information which inputted and inputted the 
user's demand information is transmitted to other communication apparatus, and the image composing transmitted from other communication 
apparatus was received according to demand information. 

[0870]In the 3rd communication apparatus of this invention, a method, and a program, Receive the demand information transmitted from other 
communication apparatus, and according to demand information, The foreground ingredient picture which consists of a foreground object ingredient 
which constitutes the foreground object of the image data which consists of a pixel value determined according to the quantity of the light which 
constitutes the picture with which it integrated in time for every pixel, The background component image which consists of a background object 
component wnich constitutes the background objects of image data is compounded, and the image composing which generated and generated image 
composing w is transmitted to other communication apparatus. 

[0871]Also in any. while becoming possible to encipher by moving to an image as a result and adding a Japanese quince, it becomes possible to 
decode by moving and removing a Japanese quince. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing; 1] It is a figure showing the composition of the 1 embodiment of the image 
processing system which applied this invention. 

[Drawing; 2] It is a figure showing the composition of the separation server of drawing 1 . 
[Drawing; 3] It is a figure showing the composition of the camera terminal device of 
drawing 1. 

[Drawing; 4] It is a figure showing the function of the separation server of drawing 1 . 
[Drawing 5] It is a figure showing the function of the separation server of drawing 1 . 
[Drawing 6] It is a figure showing the function of the motion detection server of 
drawing 1 . 

[Drawing 7] It is a figure showing the function of the motion detection server of 
drawing 1. 

[Drawing 8] It is a figure showing the function of the field specific server of drawing 1 . 
[Drawing 9] It is a figure showing the function of the field specific server of drawing 1 . 
[Drawing 10] It is a figure showing the function of the mixture ratio calculation server 
of drawing 1 . 

[Drawing 1 1] It is a figure showing the function of the mixture ratio calculation server 
of drawing 1 , 

[Drawing 12] It is a figure showing the function of the foreground background 
separation processing server of drawing 1 . 

[Drawing 13] It is a figure showing the function of the foreground background 
separation processing server of drawing 1 . 

[Drawing 14] It is a figure showing the function of the motion dotage adjustment server 
of drawing 1 . 

[Drawing 15] It is a figure showing the function of the motion dotage adjustment server 
of drawing 1 . 

[Drawing 16] It is a figure showing the function of the coding server of drawing 1 . 
[Drawing 17] It is a figure showing the function of the coding server of drawing 1 . 
[Drawing 18] It is a figure showing the function of the storage servers of drawing 1 . 
[Drawing 19] It is a figure showing the function of the storage servers of drawing 1 . 
[Drawing 20] It is a figure showing the function of the synthetic server of drawing 1 . 
[Drawing 21] It is a figure showing the function of the synthetic server of drawing 1 . 
[Drawing 22j lt is a figure showing the function of the correction server of drawing 1 . 
[Drawing 23] It is a figure showing the function of the correction server of drawing 1 . 
[Drawing 24] It is a figure showing the function of the purchase server of drawing 1 . 



[Drawing: 25] It is a figure showing the function of the sale server of drawing 1 . 

[Drawing 26] It is a figure showing the function of the retrieval server of drawing 1 . 

[Drawing 27] It is a block diagram showing a separation server. 

[Drawing 28] It is a figure explaining the image pick-up by a sensor. 

[Drawing 29] It is a figure explaining arrangement of a pixel. 

[Drawing 30] It is a figure explaining operation of a sensing element. 

[Drawing 31] It is a figure explaining the picture acquired by picturizing the object 

corresponding to the foreground currently moved, and the object corresponding to a 

stationary background. 

[Drawing 32] It is a figure explaining a background region, a foreground region, a mixing 
zone, a covered background region, and an uncovered background region. 
[Drawing 33] It is the model figure in the picture which picturized the object 
corresponding to a stationary foreground, and the object corresponding to a 
stationary background which developed the pixel value of the pixel adjacently located 
in a line with one row to the time direction. 

[Drawing 34] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 35] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 36] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 37] It is a figure showing the example which extracted the pixel of the 
foreground region, the background region, and the mixing zone. 

[Drawing 38] It is a figure showing correspondence with a pixel and the model which 
developed the pixel value to the time direction. 

[Drawing 39] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 40] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 41 ] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 42] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 43] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 44] It is a flow chart explaining processing of adjustment of the quantity of 
motion dotage. 

[Drawing 45] It is a block diagram showing an example of the composition of the field 
specific part 103. 

[Drawing 46] It is a figure explaining a picture when the object corresponding to a 



foreground is moving. 

[Drawing: 47] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing: 48] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing: 49] It is the model figure which developed the pixel value to the time direction 

and divided the period corresponding to shutter time. 

[Drawing; 50] It is a figure explaining the conditions of an area judgment. 

[Drawing; 51] It is a figure showing the example of the specific result of the field of the 

field specific part 103. 

[Drawing 52] It is a figure showing the example of the specific result of the field of the 
field specific part 103. 

[Drawing 53] It is a flow chart explaining processing of field specification. 

[Drawing 54] It is a block diagram showing other examples of the composition of the 

field specific part 103. 

[Drawing 55] It is the model figure which developed the pixel value to the time direction 

and divided the period corresponding to shutter time. 

[Drawing 56] It is a figure showing the example of a background image. 

[Drawing 57] It is a block diagram showing the composition of the binary object image 

extraction part 302. 

[Drawing 58] It is a figure explaining calculation of a correlation value. 

[Drawing 59] It is a figure explaining calculation of a correlation value. 

[Drawing 60] It is a figure showing the example of a binary object image. 

[Drawing 61] It is a block diagram showing the composition of the temporal change 

primary detecting element 303. 

[Drawing 62] It is a figure explaining the judgment of the area judgment part 342. 
[Drawing 63] It is a figure showing the example of a judgment of the temporal change 
primary detecting element 303. 

[Drawing 64] It is a flow chart explaining processing of the field specification of the 
area judgment part 103. 

[Drawing 65] It is a flow chart explaining the details of processing of an area judgment. 
[Drawing 66] It is a block diagram showing the composition of further others of the 
field specific part 103. 

[Drawing 67] It is a block diagram explaining the composition of the robust-ized part 
361. 

[Drawing 68] It is a figure explaining the motion compensation of the motion 
compensation section 381. 

[Drawing 69] It is a figure explaining the motion compensation of the motion 
compensation section 381. 

[Drawing 7Q] It is a flow chart explaining processing of field specification. 



[Drawing; 71] It is a flow chart explaining the details of processing of robust-izing. 
[Drawing; 72] It is a block diagram showing an example of the composition of the 
mixture ratio calculation part 104. 

[Drawing; 73] It is a figure showing the example which is the ideal mixture ratio alpha. 
[Drawing: 74] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing: 75] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 76] It is a figure explaining approximation using correlation of the ingredient 
of a foreground. 

[Drawing 77] It is a figure explaining the relation of C, N, and P. 

[Drawing 78] It is a block diagram showing the composition of the presumed mixture 
ratio treating part 401 . 

[Drawing 79] It is a figure showing the example of the presumed mixture ratio. 
[Drawing 80] It is a block diagram showing other composition of the mixture ratio 
calculation part 104. 

[Drawing 81] It is a flow chart explaining processing of calculation of the mixture ratio. 
[Drawing 82] It is a flow chart explaining processing of the operation of the presumed 
mixture ratio. 

[Drawing 83] the mixture ratio — it is a figure explaining the straight line which 
approximates alpha. 

[Drawing 84] the mixture ratio — it is a figure explaining the flat surface which 
approximates alpha. 

[Drawing 85] the mixture ratio — it is a figure explaining correspondence of the pixel of 
two or more frames when computing alpha. 

[Drawing 86] It is a block diagram showing other composition of the mixture ratio 
estimation processing part 401. 

[Drawing 87] It is a figure showing the example of the presumed mixture ratio. 
[Drawing 88] It is a flow chart explaining processing of mixture ratio presumption by 
the model corresponding to a covered background region. 

[Drawing 89] It is a block diagram showing an example of the composition of the 
foreground background separation part 105. 

[Drawing 90] It is a figure showing an inputted image, a foreground ingredient picture, 
and a background component image. 

[Drawing 91] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 92] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 

[Drawing 93] It is the model figure which developed the pixel value to the time direction 
and divided the period corresponding to shutter time. 



[Drawing; 94] It is a block diagram showing an example of the composition of the 
separation part 601. 

[Drawing; 95] It is a figure showing the example of the separated foreground ingredient 
picture and a background component image. 

[Drawing; 96] It is a flow chart explaining processing of separation with a foreground 
and a background. 

[Drawing; 97] It is a block diagram showing an example of the composition of the motion 
dotage controller 106. 

[Drawing 98] It is a figure explaining a batch. 

[Drawing 99] It is the model figure which developed the pixel value of the foreground 
ingredient picture to the time direction, and divided the period corresponding to 
shutter time. 

[Drawing 100] It is the model figure which developed the pixel value of the foreground 
ingredient picture to the time direction, and divided the period corresponding to 
shutter time. 

[Drawing 101] It is the model figure which developed the pixel value of the foreground 
ingredient picture to the time direction, and divided the period corresponding to 
shutter time. 

[Drawing 102] It is the model figure which developed the pixel value of the foreground 
ingredient picture to the time direction, and divided the period corresponding to 
shutter time. 

[Drawing 103] It is a figure showing other composition of the motion dotage controller 
106. 

[Drawing 104] It is a flow chart which is contained in the foreground ingredient picture 
by the motion dotage controller 106, moves, and explains processing of adjustment of 
the quantity of a Japanese quince. 

[Drawing 105] It is a block diagram showing other examples of the composition of the 
motion dotage controller 106. 

[Drawing 106] It is a figure showing the example of the model which specifies 
correspondence with a pixel value and that of the ingredient of a foreground. 
[Drawing 107] It is a figure explaining calculation of the ingredient of a foreground. 
[Drawing 108] It is a figure explaining calculation of the ingredient of a foreground. 
[Drawing 109] It is a flow chart explaining processing of removal of motion dotage of a 
foreground. 

[Drawing 1 1 0] It is a block diagram showing other composition of the function of a 
separat on server. 

[Drawing 1 1 1] It is a figure showing the composition of the synchronizer 1001. 
[Drawing 1 12] It is a block diagram showing the composition of further others of the 
function of a separation server. 

[Drawing 113] It is a block diagram showing the composition of the mixture ratio 



calculation part 1 101 . 

[Drawing; 114] It is a block diagram showing the composition of the foreground 
background separation part 1102. 

[Drawing; 1 15] It is a block diagram showing the composition of further others of the 
function of a separation server. 

[Drawing; 1 16] It is a figure showing the composition of the synchronizer 1201. 
[Drawing; 1 17] It is a flow chart explaining separation service. 
[Drawing 1 18] It is a flow chart explaining accounting. 
[Drawing 1 19] It is a figure explaining accounting. 

[Drawing 120] It is a flow chart explaining the example of others of accounting. 
[Drawing 121] It is a flow chart explaining motion detection service. 
[Drawing 122] It is a flow chart explaining field specific service. 
[Drawing 123] It is a flow chart explaining mixture ratio calculation service. 
[Drawing 124] It is a flow chart explaining foreground background separation service. 
[Drawing 125] It is a flow chart explaining motion dotage adjustment service. 
[Drawing 126] It is a figure explaining a coding server. 
[Drawing 127] It is a flow chart explaining coding service. 

[Drawing 128] It is a figure explaining the compression capability at the time of 
compression by coding processing. 

[Drawing 129] It is a figure explaining the example of others of a coding server. 

[Drawing 130] It is a flow chart explaining synthetic service. 

[Drawing 131] It is a figure explaining the motion dotage adjunct for codes. 

[Drawing 132] It is a figure explaining the motion dotage removing part for codes. 

[Drawing 133] It is a figure explaining the processing which adds the motion dotage for 

codes. 

[Drawing 134] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 135] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 136] It is a flow chart explaining encryption processing. 

[Drawing 137] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 138] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 139] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 140] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 141] It is a figure explaining the processing which adds the motion dotage for 
codes. 



[Drawing 142] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 143] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing; 144] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing; 145] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing; 146] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 147] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 148] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 149] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 150] It is a figure explaining the processing which adds the motion dotage for 
codes. 

[Drawing 151] It is a flow chart explaining the service for codes. 
[Drawing 152] It is a figure explaining a correction server. 
[Drawing 153] It is a figure explaining correction processing. 
[Drawini; 154] It is a flow chart explaining correction service. 
[Drawing 155] It is a flow chart explaining purchase service. 
[Drawing 156] It is a flow chart explaining sale service. 
[Drawing 157] It is a flow chart explaining sale accounting. 
[Drawing 158] It is a figure explaining a retrieval server. 
[Drawing 159] It is a flow chart explaining a search service. 
[Drawing 160] It is a figure explaining a search service. 
[Description of Notations] 

1 1 A separation server and 12 motion-detection server, 13 A field specific server and 
14 A mixture ratio calculation server, 15 A foreground background separation server 
and 16 A motion dotage adjustment server, 1 7 A coding server and 18 storage servers, 
19 A synthetic server and 20 A correction server and 21. A purchase server and 22 A 
sale server and 23. A retrieval server and 24 A fee collection server, 25, and 26. A 
financia server and 27 A client computer, 28 A camera terminal device and 41 CPU, 
42 ROM, 43 RAM, 46 input parts, and 47. An outputting part, 48 storage parts stores, 
and 49 The communications department and 71. CPU, 72 ROM, and 73 RAM, 76 An 
input part and 76a A sensor and 76b. GPS and 77 An outputting part and 78 A storage 
parts store, 79 The communications department, 91 magnetic disks, and 92. An optical 
disc, 93 magneto-optical discs, and 94. Semiconductor memory and 101 object 



extraction parts, 102 A motion detection part, a 103 field specific part, and 104 [ A 
frame memory, 202-1 to 202-4 / A static/dynamic detection portion, 203-1 to 203-3 
/ Area judgment part. ] A mixture ratio calculation part and 105 A foreground 
background separation part, a 106 motion dotage controller, and 107 A selecting part 
and 201 A 204 decision-flag storing frame memory, 205 A synchronizer and a 206 
decision-flag storing frame memory, 301 A background image generation part, a 302 
binary object image extraction part, and 303 [ An area judgment part and 361 / A 
robust-ized part, ] A temporal change primary detecting element and 321 Correlation 
value operation part, a 322 threshold treating part, and 341 A frame memory and 342 
381 A motion compensation section and 382 A switch, 383-1, or 383-N Frame 
memory, 384-1 thru/or a 384-N weighting section, and 385. An integrating part and 
401 A presumed mixture ratio treating part and 402. A presumed mixture ratio treating 
part and 403 A mixture ratio deciding part, 421 frame memories and 422 A frame 
memory, 423 Mixture ratio operation part and 441 A selecting part and 442. A 
presumed mixture ratio treating part and 443 A presumed mixture ratio treating part, 
444 A selecting part and 501 A delay circuit and 502. It adds and is a lump part and 
503. Operation part and 601 A separation part, 602 A switch and 603 A synchronizer 
and 604. A switch and 605 A synchronizer and 621 A frame memory, 622 A separation 
block and 623 A frame memory, 631 An uncovered region processing part and 632. A 
covered region processing part and 633 A synchronizer, 634 synchronizers and 801 A 
batch deciding part, 802 A modeling part and 803 An equation generation part, It adds 
804 anc is [ A batch deciding part, 902 modeling parts, and 903 / An equation 
generation part and 904 / Operation part and 905 / An amendment part, ] a lump part 
and 805. Operation part, an 806 motion dotage adjunct, and 807 A selecting part and 
821 A selecting part and 901 906 A motion dotage adjunct and 907 A selecting part, 
1001 synchronizers, and 1021 A background component generation part and 102. 2 A 
mixing zone image synthesis section and 1023 An image synthesis section, 1101 A 
mixture ratio calculation part and 1 102 A foreground background separation part, 1 121 
A selecting part and 1201 A synchronizer and 1221. A selecting part and 2001 A 
coding part and 2002. Separation sections and 2021 The motion dotage adjunct for 
codes, and 2031. The input communications department and 2032 An imaging part and 
2033. A motion dotage adjunct and 2041 The motion dotage removing part for codes, 
2051 The input communications department and 2052 A motion dotage removing part, 
2053 A signal conversion section and 2101 [A storage parts store and 2164 / A 
demand information generating part and 2165 / The communications department and 
2166 / Time operation part and 2167 / Separation sections ] A motion dotage 
controller, 2102 synchronizers, and 2161 A control section and 2162 A database and 
2163 



